2024-03-25T16:10:38.863 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:10:38.864 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_amathuri@teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --verbose --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256 -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256/orig.config.yaml 2024-03-25T16:10:38.916 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:10:39.006 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621256' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi063.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCuqPn9P/7bTYjv4OwFl5It97HHvWJqNzhrbIGO9jXyoSczazWUGovbwr+XcXWyTjBXQq5Rnxziuri9JqQFfy2c= smithi081.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBfBt3+3t0WojqxorpmvGW+mvVvJwbVcKMoUxRrsp1ZH3dMxNuK0Q3eqzhXGLQyLCJEcwaE+pFQXqyAv0TkZGFw= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:10:39.006 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:10:39.007 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:10:39.007 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:10:39.009 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:10:39.033 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:10:39.033 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:10:39.033 INFO:teuthology.packaging:ref: None 2024-03-25T16:10:39.033 INFO:teuthology.packaging:tag: None 2024-03-25T16:10:39.033 INFO:teuthology.packaging:branch: main 2024-03-25T16:10:39.033 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:10:39.034 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:10:39.371 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:10:39.373 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:10:39.379 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:10:39.379 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:10:39.394 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:10:39.413 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:10:39.419 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:10:39.444 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:05:36.330033', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCuqPn9P/7bTYjv4OwFl5It97HHvWJqNzhrbIGO9jXyoSczazWUGovbwr+XcXWyTjBXQq5Rnxziuri9JqQFfy2c='} 2024-03-25T16:10:39.471 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:05:36.328710', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBfBt3+3t0WojqxorpmvGW+mvVvJwbVcKMoUxRrsp1ZH3dMxNuK0Q3eqzhXGLQyLCJEcwaE+pFQXqyAv0TkZGFw='} 2024-03-25T16:10:39.471 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:10:39.478 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:10:39.478 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:10:39.478 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:10:39.558 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcd4b7660d0>, signals=[15]) 2024-03-25T16:10:39.559 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:10:39.566 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:10:39.566 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T16:10:39.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:10:39.721 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T16:10:39.722 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:10:39.814 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:10:39.819 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2024-03-25T16:10:39.837 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2024-03-25T16:10:39.838 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2024-03-25T16:10:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="CentOS Stream" 2024-03-25T16:10:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="9" 2024-03-25T16:10:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:ID="centos" 2024-03-25T16:10:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:10:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="9" 2024-03-25T16:10:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:10:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:10:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:ANSI_COLOR="0;31" 2024-03-25T16:10:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:10:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:10:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:10:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:10:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:10:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:10:39.897 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2024-03-25T16:10:39.928 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2024-03-25T16:10:39.945 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2024-03-25T16:10:39.946 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2024-03-25T16:10:40.001 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="CentOS Stream" 2024-03-25T16:10:40.001 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="9" 2024-03-25T16:10:40.001 INFO:teuthology.orchestra.run.smithi081.stdout:ID="centos" 2024-03-25T16:10:40.001 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:10:40.001 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="9" 2024-03-25T16:10:40.001 INFO:teuthology.orchestra.run.smithi081.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:10:40.001 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:10:40.001 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2024-03-25T16:10:40.001 INFO:teuthology.orchestra.run.smithi081.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:10:40.001 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:10:40.001 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:10:40.001 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:10:40.002 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:10:40.002 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:10:40.002 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2024-03-25T16:10:40.028 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:10:40.036 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:10:40.042 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:10:40.042 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:10:40.045 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:10:40.061 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:10:40.069 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:10:40.070 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:10:40.103 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:10:40.208 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:10:40.262 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:10:40.276 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:10:40.276 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:10:40.276 DEBUG:teuthology.orchestra.run.smithi063:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:10:40.277 DEBUG:teuthology.orchestra.run.smithi081:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:10:40.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:10:40.294 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-03-25T16:10:40.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:10:40.295 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-03-25T16:10:40.352 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:10:40.353 INFO:teuthology.task.kernel:Running kernel on smithi081: 5.14.0-361.el9.x86_64 2024-03-25T16:10:40.353 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum install -y kernel 2024-03-25T16:10:40.354 INFO:teuthology.orchestra.run.smithi063.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:10:40.354 INFO:teuthology.task.kernel:Running kernel on smithi063: 5.14.0-361.el9.x86_64 2024-03-25T16:10:40.354 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum install -y kernel 2024-03-25T16:10:42.639 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T16:10:42.671 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS-9 - AppStream 9.8 MB/s | 6.7 MB 00:00 2024-03-25T16:10:44.373 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2024-03-25T16:10:44.504 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS-9 - BaseOS 2.6 MB/s | 2.0 MB 00:00 2024-03-25T16:10:45.155 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-25T16:10:45.436 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-25T16:10:46.939 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-25T16:10:47.429 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-25T16:10:50.782 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-25T16:10:51.380 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-25T16:10:52.216 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-25T16:10:53.448 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 16 kB 00:00 2024-03-25T16:10:55.101 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 21 MB 00:02 2024-03-25T16:10:55.224 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-25T16:11:00.309 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 4.9 kB/s | 1.7 kB 00:00 2024-03-25T16:11:00.315 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-25T16:11:02.391 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:11:02.391 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:11:02.490 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:11:02.500 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:11:02.500 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-03-25T16:11:02.500 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:11:02.500 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:11:02.500 INFO:teuthology.orchestra.run.smithi081.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:11:02.500 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T16:11:02.500 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:11:02.500 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:11:02.500 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:11:02.500 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:11:02.501 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:11:02.501 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:11:02.501 INFO:teuthology.orchestra.run.smithi081.stdout:Install 4 Packages 2024-03-25T16:11:02.501 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:11:02.501 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 76 M 2024-03-25T16:11:02.501 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 123 M 2024-03-25T16:11:02.501 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:11:02.531 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:11:02.531 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:11:02.626 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:11:02.636 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:11:02.636 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-03-25T16:11:02.636 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:11:02.636 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:11:02.636 INFO:teuthology.orchestra.run.smithi063.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:11:02.636 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T16:11:02.636 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:11:02.636 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:11:02.636 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:11:02.636 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:11:02.636 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:11:02.637 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:11:02.637 INFO:teuthology.orchestra.run.smithi063.stdout:Install 4 Packages 2024-03-25T16:11:02.637 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:11:02.637 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 76 M 2024-03-25T16:11:02.637 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 123 M 2024-03-25T16:11:02.637 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:11:02.777 INFO:teuthology.orchestra.run.smithi081.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 189 kB/s | 44 kB 00:00 2024-03-25T16:11:02.975 INFO:teuthology.orchestra.run.smithi063.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 291 kB/s | 44 kB 00:00 2024-03-25T16:11:04.377 INFO:teuthology.orchestra.run.smithi081.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 8.1 MB/s | 15 MB 00:01 2024-03-25T16:11:05.269 INFO:teuthology.orchestra.run.smithi081.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 12 MB/s | 33 MB 00:02 2024-03-25T16:11:06.217 INFO:teuthology.orchestra.run.smithi063.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 4.4 MB/s | 15 MB 00:03 2024-03-25T16:11:07.444 INFO:teuthology.orchestra.run.smithi081.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 5.9 MB/s | 27 MB 00:04 2024-03-25T16:11:07.445 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:11:07.445 INFO:teuthology.orchestra.run.smithi081.stdout:Total 15 MB/s | 76 MB 00:04 2024-03-25T16:11:07.733 INFO:teuthology.orchestra.run.smithi063.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 5.8 MB/s | 27 MB 00:04 2024-03-25T16:11:08.106 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:11:08.464 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:11:08.464 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:11:08.892 INFO:teuthology.orchestra.run.smithi063.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 5.5 MB/s | 33 MB 00:06 2024-03-25T16:11:08.892 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:11:08.892 INFO:teuthology.orchestra.run.smithi063.stdout:Total 12 MB/s | 76 MB 00:06 2024-03-25T16:11:09.501 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:11:09.501 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:11:09.583 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:11:10.005 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:11:10.006 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:11:11.056 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:11:11.056 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:11:11.951 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:11:12.548 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:11:12.637 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:13.669 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:11:14.286 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:11:14.383 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:15.377 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:15.508 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:17.246 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:17.385 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:20.191 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:20.308 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:22.455 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:22.493 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:25.031 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:27.587 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:54.240 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:54.241 INFO:teuthology.orchestra.run.smithi081.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:11:54.241 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:11:54.285 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:54.671 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:54.671 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:11:54.672 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:54.672 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:55.101 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:55.101 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:11:55.101 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:11:55.101 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:11:55.101 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:11:55.101 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:11:55.101 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:11:55.101 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:11:55.101 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:11:55.428 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:11:55.567 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:11:55.568 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:11:55.568 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:11:55.569 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:11:55.569 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:11:55.569 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:11:55.569 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:11:55.569 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:11:55.647 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:11:55.648 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:11:55.648 DEBUG:teuthology.orchestra.run.smithi081:> sudo rpm -qi grub2-tools 2024-03-25T16:11:55.682 INFO:teuthology.orchestra.run.smithi081.stdout:Name : grub2-tools 2024-03-25T16:11:55.682 INFO:teuthology.orchestra.run.smithi081.stdout:Epoch : 1 2024-03-25T16:11:55.682 INFO:teuthology.orchestra.run.smithi081.stdout:Version : 2.06 2024-03-25T16:11:55.682 INFO:teuthology.orchestra.run.smithi081.stdout:Release : 68.el9 2024-03-25T16:11:55.682 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: x86_64 2024-03-25T16:11:55.682 INFO:teuthology.orchestra.run.smithi081.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:11:55.682 INFO:teuthology.orchestra.run.smithi081.stdout:Group : Unspecified 2024-03-25T16:11:55.683 INFO:teuthology.orchestra.run.smithi081.stdout:Size : 8264254 2024-03-25T16:11:55.683 INFO:teuthology.orchestra.run.smithi081.stdout:License : GPLv3+ 2024-03-25T16:11:55.683 INFO:teuthology.orchestra.run.smithi081.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:11:55.683 INFO:teuthology.orchestra.run.smithi081.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:11:55.683 INFO:teuthology.orchestra.run.smithi081.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:11:55.683 INFO:teuthology.orchestra.run.smithi081.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:11:55.683 INFO:teuthology.orchestra.run.smithi081.stdout:Packager : builder@centos.org 2024-03-25T16:11:55.683 INFO:teuthology.orchestra.run.smithi081.stdout:Vendor : CentOS 2024-03-25T16:11:55.683 INFO:teuthology.orchestra.run.smithi081.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:11:55.683 INFO:teuthology.orchestra.run.smithi081.stdout:Summary : Support tools for GRUB. 2024-03-25T16:11:55.683 INFO:teuthology.orchestra.run.smithi081.stdout:Description : 2024-03-25T16:11:55.683 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:11:55.683 INFO:teuthology.orchestra.run.smithi081.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:11:55.684 INFO:teuthology.orchestra.run.smithi081.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:11:55.684 INFO:teuthology.orchestra.run.smithi081.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:11:55.684 INFO:teuthology.orchestra.run.smithi081.stdout:hardware devices. 2024-03-25T16:11:55.684 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:11:55.684 INFO:teuthology.orchestra.run.smithi081.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:11:55.685 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:11:55.685 INFO:teuthology.task.kernel:Updating grub on smithi081 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:11:55.685 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:11:56.338 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2024-03-25T16:11:57.823 INFO:teuthology.orchestra.run.smithi081.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:11:57.857 INFO:teuthology.orchestra.run.smithi081.stderr:done 2024-03-25T16:11:57.859 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-03-25T16:11:57.874 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.xjL0en1VtE 2024-03-25T16:11:57.875 DEBUG:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xjL0en1VtE 2024-03-25T16:11:57.946 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.xjL0en1VtE 2024-03-25T16:11:58.095 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.xjL0en1VtE is 6KB 2024-03-25T16:11:58.143 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.xjL0en1VtE 2024-03-25T16:11:58.156 DEBUG:teuthology.orchestra.run.smithi081:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:11:58.221 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:11:58.221 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:11:58.221 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:11:58.221 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:11:58.222 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:11:58.358 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2024-03-25T16:11:58.368 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:58.368 INFO:teuthology.orchestra.run.smithi063.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:11:58.368 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:11:58.419 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:58.720 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:58.720 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:11:58.720 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:58.720 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:59.129 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:59.129 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:11:59.129 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:11:59.129 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:11:59.129 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:11:59.129 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:11:59.129 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:11:59.130 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:11:59.130 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:11:59.459 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:11:59.575 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:11:59.576 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:11:59.576 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:11:59.576 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:11:59.576 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:11:59.576 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:11:59.576 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi063.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:11:59.576 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:11:59.659 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:11:59.659 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:11:59.659 DEBUG:teuthology.orchestra.run.smithi063:> sudo rpm -qi grub2-tools 2024-03-25T16:11:59.696 INFO:teuthology.orchestra.run.smithi063.stdout:Name : grub2-tools 2024-03-25T16:11:59.697 INFO:teuthology.orchestra.run.smithi063.stdout:Epoch : 1 2024-03-25T16:11:59.697 INFO:teuthology.orchestra.run.smithi063.stdout:Version : 2.06 2024-03-25T16:11:59.697 INFO:teuthology.orchestra.run.smithi063.stdout:Release : 68.el9 2024-03-25T16:11:59.697 INFO:teuthology.orchestra.run.smithi063.stdout:Architecture: x86_64 2024-03-25T16:11:59.697 INFO:teuthology.orchestra.run.smithi063.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:11:59.697 INFO:teuthology.orchestra.run.smithi063.stdout:Group : Unspecified 2024-03-25T16:11:59.697 INFO:teuthology.orchestra.run.smithi063.stdout:Size : 8264254 2024-03-25T16:11:59.697 INFO:teuthology.orchestra.run.smithi063.stdout:License : GPLv3+ 2024-03-25T16:11:59.697 INFO:teuthology.orchestra.run.smithi063.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:11:59.697 INFO:teuthology.orchestra.run.smithi063.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:11:59.697 INFO:teuthology.orchestra.run.smithi063.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:11:59.697 INFO:teuthology.orchestra.run.smithi063.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:11:59.697 INFO:teuthology.orchestra.run.smithi063.stdout:Packager : builder@centos.org 2024-03-25T16:11:59.697 INFO:teuthology.orchestra.run.smithi063.stdout:Vendor : CentOS 2024-03-25T16:11:59.698 INFO:teuthology.orchestra.run.smithi063.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:11:59.698 INFO:teuthology.orchestra.run.smithi063.stdout:Summary : Support tools for GRUB. 2024-03-25T16:11:59.698 INFO:teuthology.orchestra.run.smithi063.stdout:Description : 2024-03-25T16:11:59.698 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:11:59.698 INFO:teuthology.orchestra.run.smithi063.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:11:59.698 INFO:teuthology.orchestra.run.smithi063.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:11:59.698 INFO:teuthology.orchestra.run.smithi063.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:11:59.698 INFO:teuthology.orchestra.run.smithi063.stdout:hardware devices. 2024-03-25T16:11:59.698 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:11:59.698 INFO:teuthology.orchestra.run.smithi063.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:11:59.699 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:11:59.699 INFO:teuthology.task.kernel:Updating grub on smithi063 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:11:59.699 DEBUG:teuthology.orchestra.run.smithi063:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:12:00.313 INFO:teuthology.orchestra.run.smithi063.stderr:Generating grub configuration file ... 2024-03-25T16:12:01.962 INFO:teuthology.orchestra.run.smithi063.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:12:01.980 INFO:teuthology.orchestra.run.smithi063.stderr:done 2024-03-25T16:12:01.982 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-03-25T16:12:01.997 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.FXOkHBqSzS 2024-03-25T16:12:01.998 DEBUG:teuthology.orchestra.run.smithi063:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FXOkHBqSzS 2024-03-25T16:12:02.071 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.FXOkHBqSzS 2024-03-25T16:12:02.216 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.FXOkHBqSzS is 6KB 2024-03-25T16:12:02.266 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.FXOkHBqSzS 2024-03-25T16:12:02.280 DEBUG:teuthology.orchestra.run.smithi063:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:12:02.349 INFO:teuthology.orchestra.run.smithi063.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:12:02.349 INFO:teuthology.orchestra.run.smithi063.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:12:02.349 INFO:teuthology.orchestra.run.smithi063.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:12:02.349 INFO:teuthology.orchestra.run.smithi063.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:12:02.350 DEBUG:teuthology.orchestra.run.smithi063:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:12:02.507 DEBUG:teuthology.orchestra.run.smithi063:> sudo shutdown -r now 2024-03-25T16:12:28.383 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:12:28.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-25T16:12:28.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:12:32.510 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:12:32.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-25T16:12:32.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:12:46.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-03-25T16:12:50.846 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-03-25T16:12:55.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-25T16:12:55.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:12:59.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-25T16:12:59.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:02.914 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-03-25T16:13:14.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-25T16:13:14.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:33.346 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-03-25T16:13:48.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-25T16:13:48.362 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:48.715 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-03-25T16:13:49.151 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-25T16:13:49.151 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:13:49.151 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-03-25T16:13:49.165 INFO:teuthology.orchestra.run.smithi063.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:13:49.165 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:13:49.165 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:13:49.165 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:13:50.166 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:13:50.167 DEBUG:teuthology.orchestra.run.smithi063:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:13:50.224 INFO:teuthology.orchestra.run.smithi063.stdout:ttyS1 2024-03-25T16:13:50.272 DEBUG:teuthology.parallel:result is None 2024-03-25T16:13:55.757 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:14:07.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-25T16:14:07.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:08.309 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-03-25T16:14:08.713 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-25T16:14:08.713 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:14:08.713 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-03-25T16:14:08.729 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:14:08.729 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:14:08.729 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:14:08.730 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:14:09.730 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:14:09.730 DEBUG:teuthology.orchestra.run.smithi081:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:14:09.820 INFO:teuthology.orchestra.run.smithi081.stdout:ttyS1 2024-03-25T16:14:09.852 DEBUG:teuthology.parallel:result is None 2024-03-25T16:14:09.852 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:14:09.859 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:14:09.859 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:14:09.863 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:14:09.880 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:14:09.887 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:14:09.897 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:14:09.897 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:14:09.920 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:14:09.981 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:14:09.988 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:14:09.988 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:14:09.991 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:14:10.021 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:14:10.027 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:14:10.035 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:14:10.041 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:14:10.043 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:14:10.050 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:14:10.050 DEBUG:teuthology.orchestra.run.smithi063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:14:10.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:14:10.113 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:14:10.122 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:14:10.122 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:14:10.148 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:14:10.169 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:14:10.245 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:14:10.291 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:14:10.373 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:14:10.417 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T16:14:10.417 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:14:10.484 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:14:10.528 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:14:10.583 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:14:10.670 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:14:10.711 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T16:14:10.711 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:14:10.779 DEBUG:teuthology.orchestra.run.smithi063:> sudo service rsyslog restart 2024-03-25T16:14:10.781 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2024-03-25T16:14:10.827 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:14:10.876 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:14:11.335 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:14:11.343 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:14:11.343 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:14:11.360 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:14:11.368 DEBUG:teuthology.orchestra.run.smithi063:> sudo service auditd rotate 2024-03-25T16:14:11.419 INFO:teuthology.orchestra.run.smithi063.stdout:Rotating logs: 2024-03-25T16:14:11.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo service auditd rotate 2024-03-25T16:14:11.489 INFO:teuthology.orchestra.run.smithi081.stdout:Rotating logs: 2024-03-25T16:14:11.491 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:14:11.491 DEBUG:teuthology.orchestra.run.smithi063:> /usr/sbin/getenforce 2024-03-25T16:14:11.517 INFO:teuthology.orchestra.run.smithi063.stdout:Permissive 2024-03-25T16:14:11.518 DEBUG:teuthology.orchestra.run.smithi081:> /usr/sbin/getenforce 2024-03-25T16:14:11.540 INFO:teuthology.orchestra.run.smithi081.stdout:Permissive 2024-03-25T16:14:11.540 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi063.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi081.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:14:11.541 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:14:11.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:14:11.587 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:14:11.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:14:11.616 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:14:11.616 DEBUG:teuthology.orchestra.run.smithi063:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:14:11.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:14:11.709 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:14:11.719 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:14:11.737 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:14:11.752 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi063.front.sepia.ceph.com,smithi081.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:18:30.772 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi063.front.sepia.ceph.com'), Remote(name='ubuntu@smithi081.front.sepia.ceph.com')] 2024-03-25T16:18:30.773 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-25T16:18:30.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:18:30.870 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-03-25T16:18:30.940 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-25T16:18:30.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-25T16:18:30.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:18:31.031 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-03-25T16:18:31.117 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-25T16:18:31.118 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:18:31.129 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:18:31.129 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:18:31.130 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:18:31.132 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:18:31.132 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:18:31.160 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:18:31.176 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:18:31.189 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:18:31.204 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: ntpd: command not found 2024-03-25T16:18:31.206 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:18:31.218 INFO:teuthology.orchestra.run.smithi063.stdout:506 Cannot talk to daemon 2024-03-25T16:18:31.235 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:18:31.235 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: ntpd: command not found 2024-03-25T16:18:31.249 INFO:teuthology.orchestra.run.smithi081.stdout:506 Cannot talk to daemon 2024-03-25T16:18:31.251 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:18:31.266 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:18:31.282 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:18:31.299 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:18:31.303 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:18:31.303 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-03-25T16:18:31.303 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:31.303 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:31.303 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:31.303 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:31.340 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:18:31.343 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:18:31.343 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-03-25T16:18:31.343 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:31.343 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:31.343 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:31.343 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:31.345 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:18:31.353 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:18:31.353 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:18:31.353 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:18:31.354 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:18:31.361 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:18:31.361 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:18:31.361 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:18:31.363 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:18:31.366 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:18:31.366 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:18:31.515 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:18:31.516 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:18:31.581 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:18:31.581 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:18:31.877 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:18:31.878 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T16:18:31.878 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:18:31.894 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:18:31.894 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T16:18:31.895 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:18:31.905 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:18:31.905 DEBUG:teuthology.orchestra.run.smithi063:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:18:31.922 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:18:31.923 DEBUG:teuthology.orchestra.run.smithi081:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:18:31.974 DEBUG:teuthology.orchestra.run.smithi063:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:18:31.991 DEBUG:teuthology.orchestra.run.smithi081:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:18:32.053 DEBUG:teuthology.orchestra.run.smithi063:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:18:32.070 DEBUG:teuthology.orchestra.run.smithi081:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:18:32.120 INFO:teuthology.orchestra.run.smithi063.stdout:check_obsoletes = 1 2024-03-25T16:18:32.121 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum clean all 2024-03-25T16:18:32.136 INFO:teuthology.orchestra.run.smithi081.stdout:check_obsoletes = 1 2024-03-25T16:18:32.138 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum clean all 2024-03-25T16:18:32.358 INFO:teuthology.orchestra.run.smithi063.stdout:74 files removed 2024-03-25T16:18:32.380 INFO:teuthology.orchestra.run.smithi081.stdout:74 files removed 2024-03-25T16:18:32.385 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-radosgw 2024-03-25T16:18:32.409 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-radosgw 2024-03-25T16:18:33.556 INFO:teuthology.orchestra.run.smithi063.stdout:ceph packages for x86_64 95 kB/s | 82 kB 00:00 2024-03-25T16:18:33.936 INFO:teuthology.orchestra.run.smithi081.stdout:ceph packages for x86_64 67 kB/s | 82 kB 00:01 2024-03-25T16:18:34.331 INFO:teuthology.orchestra.run.smithi063.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-25T16:18:34.753 INFO:teuthology.orchestra.run.smithi081.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T16:18:35.148 INFO:teuthology.orchestra.run.smithi063.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T16:18:35.437 INFO:teuthology.orchestra.run.smithi081.stdout:ceph source packages 2.9 kB/s | 1.9 kB 00:00 2024-03-25T16:18:35.676 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T16:18:35.873 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-25T16:18:37.101 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS-9 - BaseOS 7.7 MB/s | 2.0 MB 00:00 2024-03-25T16:18:37.281 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-25T16:18:38.433 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 8.1 MB/s | 8.0 MB 00:00 2024-03-25T16:18:38.689 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 7.5 MB/s | 8.0 MB 00:01 2024-03-25T16:18:40.693 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-25T16:18:40.726 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-25T16:18:45.086 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 8.1 MB/s | 6.1 MB 00:00 2024-03-25T16:18:45.166 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 5.9 MB/s | 6.1 MB 00:01 2024-03-25T16:18:46.709 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-25T16:18:46.970 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-03-25T16:18:47.980 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-25T16:18:48.438 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-25T16:18:53.239 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-25T16:18:53.252 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-25T16:18:55.679 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:18:55.682 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:18:55.683 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T16:18:55.683 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:18:55.683 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:18:55.683 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:18:55.683 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrading: 2024-03-25T16:18:55.683 INFO:teuthology.orchestra.run.smithi081.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:18:55.683 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:18:55.683 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T16:18:55.683 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options 2024-03-25T16:18:55.683 INFO:teuthology.orchestra.run.smithi081.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:18:55.683 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:18:55.683 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:18:55.684 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:18:55.684 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:18:55.684 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:18:55.684 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:18:55.684 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:18:55.684 INFO:teuthology.orchestra.run.smithi081.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:18:55.684 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:18:55.684 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:18:55.684 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:18:55.684 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:18:55.684 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:18:55.684 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:18:55.685 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:18:55.685 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:18:55.685 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse 2024-03-25T16:18:55.685 INFO:teuthology.orchestra.run.smithi081.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:18:55.685 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:18:55.685 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:18:55.685 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:18:55.685 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:18:55.685 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:18:55.685 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:18:55.685 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:18:55.685 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:18:55.685 INFO:teuthology.orchestra.run.smithi081.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:18:55.686 INFO:teuthology.orchestra.run.smithi081.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:18:55.686 INFO:teuthology.orchestra.run.smithi081.stdout:Installing weak dependencies: 2024-03-25T16:18:55.686 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:18:55.686 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:18:55.686 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:18:55.686 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:18:55.686 INFO:teuthology.orchestra.run.smithi081.stdout:Install 30 Packages 2024-03-25T16:18:55.686 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrade 2 Packages 2024-03-25T16:18:55.686 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:18:55.687 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 58 M 2024-03-25T16:18:55.687 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:18:55.761 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:18:55.777 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:18:55.777 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T16:18:55.777 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:18:55.777 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:18:55.777 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:18:55.777 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrading: 2024-03-25T16:18:55.777 INFO:teuthology.orchestra.run.smithi063.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:18:55.777 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:18:55.777 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T16:18:55.777 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options 2024-03-25T16:18:55.777 INFO:teuthology.orchestra.run.smithi063.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:18:55.778 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:18:55.778 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:18:55.778 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:18:55.778 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:18:55.778 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:18:55.778 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:18:55.778 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:18:55.778 INFO:teuthology.orchestra.run.smithi063.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:18:55.778 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:18:55.778 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:18:55.778 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:18:55.778 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:18:55.778 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:18:55.778 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:18:55.779 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:18:55.779 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:18:55.779 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse 2024-03-25T16:18:55.779 INFO:teuthology.orchestra.run.smithi063.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:18:55.779 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:18:55.779 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:18:55.779 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:18:55.779 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:18:55.779 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:18:55.779 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:18:55.779 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:18:55.779 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:18:55.779 INFO:teuthology.orchestra.run.smithi063.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:18:55.779 INFO:teuthology.orchestra.run.smithi063.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:18:55.780 INFO:teuthology.orchestra.run.smithi063.stdout:Installing weak dependencies: 2024-03-25T16:18:55.780 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:18:55.780 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:18:55.780 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:18:55.780 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:18:55.780 INFO:teuthology.orchestra.run.smithi063.stdout:Install 30 Packages 2024-03-25T16:18:55.780 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrade 2 Packages 2024-03-25T16:18:55.780 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:18:55.781 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 58 M 2024-03-25T16:18:55.781 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:19:01.619 INFO:teuthology.orchestra.run.smithi081.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 5.3 MB 00:05 2024-03-25T16:19:01.711 INFO:teuthology.orchestra.run.smithi081.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 275 kB/s | 25 kB 00:00 2024-03-25T16:19:02.006 INFO:teuthology.orchestra.run.smithi063.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 903 kB/s | 5.3 MB 00:06 2024-03-25T16:19:02.098 INFO:teuthology.orchestra.run.smithi063.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 275 kB/s | 25 kB 00:00 2024-03-25T16:19:02.545 INFO:teuthology.orchestra.run.smithi081.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 890 kB/s | 742 kB 00:00 2024-03-25T16:19:02.748 INFO:teuthology.orchestra.run.smithi063.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.1 MB/s | 742 kB 00:00 2024-03-25T16:19:03.112 INFO:teuthology.orchestra.run.smithi081.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 880 kB/s | 498 kB 00:00 2024-03-25T16:19:03.165 INFO:teuthology.orchestra.run.smithi063.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.2 MB/s | 498 kB 00:00 2024-03-25T16:19:07.090 INFO:teuthology.orchestra.run.smithi063.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 1.3 MB/s | 5.1 MB 00:03 2024-03-25T16:19:07.182 INFO:teuthology.orchestra.run.smithi063.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 494 kB/s | 45 kB 00:00 2024-03-25T16:19:07.349 INFO:teuthology.orchestra.run.smithi063.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 788 kB/s | 131 kB 00:00 2024-03-25T16:19:07.516 INFO:teuthology.orchestra.run.smithi063.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 972 kB/s | 162 kB 00:00 2024-03-25T16:19:07.661 INFO:teuthology.orchestra.run.smithi081.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 1.1 MB/s | 5.1 MB 00:04 2024-03-25T16:19:07.749 INFO:teuthology.orchestra.run.smithi063.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 1.3 MB/s | 320 kB 00:00 2024-03-25T16:19:07.762 INFO:teuthology.orchestra.run.smithi081.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 452 kB/s | 45 kB 00:00 2024-03-25T16:19:07.945 INFO:teuthology.orchestra.run.smithi081.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 716 kB/s | 131 kB 00:00 2024-03-25T16:19:08.058 INFO:teuthology.orchestra.run.smithi063.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 965 kB/s | 298 kB 00:00 2024-03-25T16:19:08.087 INFO:teuthology.orchestra.run.smithi081.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.1 MB/s | 162 kB 00:00 2024-03-25T16:19:08.192 INFO:teuthology.orchestra.run.smithi063.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 746 kB/s | 99 kB 00:00 2024-03-25T16:19:08.329 INFO:teuthology.orchestra.run.smithi081.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 1.3 MB/s | 320 kB 00:00 2024-03-25T16:19:08.434 INFO:teuthology.orchestra.run.smithi063.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 438 kB/s | 106 kB 00:00 2024-03-25T16:19:08.551 INFO:teuthology.orchestra.run.smithi063.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-25T16:19:08.563 INFO:teuthology.orchestra.run.smithi081.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 298 kB 00:00 2024-03-25T16:19:08.593 INFO:teuthology.orchestra.run.smithi063.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T16:19:08.705 INFO:teuthology.orchestra.run.smithi081.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 702 kB/s | 99 kB 00:00 2024-03-25T16:19:08.918 INFO:teuthology.orchestra.run.smithi063.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.0 MB/s | 661 kB 00:00 2024-03-25T16:19:08.963 INFO:teuthology.orchestra.run.smithi081.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 410 kB/s | 106 kB 00:00 2024-03-25T16:19:09.076 INFO:teuthology.orchestra.run.smithi063.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 282 kB 00:00 2024-03-25T16:19:09.097 INFO:teuthology.orchestra.run.smithi081.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-03-25T16:19:09.135 INFO:teuthology.orchestra.run.smithi063.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 706 kB/s | 41 kB 00:00 2024-03-25T16:19:09.156 INFO:teuthology.orchestra.run.smithi081.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 758 kB/s | 44 kB 00:00 2024-03-25T16:19:09.281 INFO:teuthology.orchestra.run.smithi081.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 661 kB 00:00 2024-03-25T16:19:09.365 INFO:teuthology.orchestra.run.smithi081.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-25T16:19:09.423 INFO:teuthology.orchestra.run.smithi081.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 708 kB/s | 41 kB 00:00 2024-03-25T16:19:09.532 INFO:teuthology.orchestra.run.smithi081.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 2.0 MB/s | 216 kB 00:00 2024-03-25T16:19:09.582 INFO:teuthology.orchestra.run.smithi081.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 699 kB/s | 35 kB 00:00 2024-03-25T16:19:09.724 INFO:teuthology.orchestra.run.smithi081.stdout:(20/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 736 kB/s | 9.7 MB 00:13 2024-03-25T16:19:09.749 INFO:teuthology.orchestra.run.smithi081.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 267 kB/s | 44 kB 00:00 2024-03-25T16:19:09.917 INFO:teuthology.orchestra.run.smithi081.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-25T16:19:09.959 INFO:teuthology.orchestra.run.smithi081.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 654 kB/s | 26 kB 00:00 2024-03-25T16:19:10.025 INFO:teuthology.orchestra.run.smithi081.stdout:(24/32): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-03-25T16:19:10.051 INFO:teuthology.orchestra.run.smithi081.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 544 kB/s | 49 kB 00:00 2024-03-25T16:19:10.069 INFO:teuthology.orchestra.run.smithi063.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 232 kB/s | 216 kB 00:00 2024-03-25T16:19:10.076 INFO:teuthology.orchestra.run.smithi081.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-25T16:19:10.118 INFO:teuthology.orchestra.run.smithi081.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-25T16:19:10.143 INFO:teuthology.orchestra.run.smithi081.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-25T16:19:10.294 INFO:teuthology.orchestra.run.smithi081.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-25T16:19:10.785 INFO:teuthology.orchestra.run.smithi063.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 49 kB/s | 35 kB 00:00 2024-03-25T16:19:11.528 INFO:teuthology.orchestra.run.smithi063.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 60 kB/s | 44 kB 00:00 2024-03-25T16:19:12.028 INFO:teuthology.orchestra.run.smithi063.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 618 kB/s | 309 kB 00:00 2024-03-25T16:19:13.035 INFO:teuthology.orchestra.run.smithi081.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 1.2 MB/s | 3.4 MB 00:02 2024-03-25T16:19:13.053 INFO:teuthology.orchestra.run.smithi063.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 4.4 MB 00:01 2024-03-25T16:19:13.320 INFO:teuthology.orchestra.run.smithi063.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 97 kB/s | 26 kB 00:00 2024-03-25T16:19:13.670 INFO:teuthology.orchestra.run.smithi063.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 139 kB/s | 49 kB 00:00 2024-03-25T16:19:14.012 INFO:teuthology.orchestra.run.smithi063.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 197 kB/s | 67 kB 00:00 2024-03-25T16:19:14.060 INFO:teuthology.orchestra.run.smithi081.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 851 kB/s | 3.1 MB 00:03 2024-03-25T16:19:14.121 INFO:teuthology.orchestra.run.smithi063.stdout:(26/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 549 kB/s | 9.7 MB 00:18 2024-03-25T16:19:14.638 INFO:teuthology.orchestra.run.smithi063.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 371 kB/s | 192 kB 00:00 2024-03-25T16:19:14.788 INFO:teuthology.orchestra.run.smithi063.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.1 MB/s | 838 kB 00:00 2024-03-25T16:19:15.397 INFO:teuthology.orchestra.run.smithi063.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2024-03-25T16:19:17.547 INFO:teuthology.orchestra.run.smithi063.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 1.2 MB/s | 3.4 MB 00:02 2024-03-25T16:19:21.835 INFO:teuthology.orchestra.run.smithi081.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 794 kB/s | 20 MB 00:25 2024-03-25T16:19:21.841 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:21.841 INFO:teuthology.orchestra.run.smithi081.stdout:Total 2.2 MB/s | 58 MB 00:26 2024-03-25T16:19:21.873 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:19:21.911 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:19:21.911 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:19:22.385 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:19:22.385 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:19:23.096 INFO:teuthology.orchestra.run.smithi063.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 416 kB/s | 3.1 MB 00:07 2024-03-25T16:19:23.873 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:19:24.099 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:19:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:19:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:19:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:19:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:19:24.485 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:19:24.547 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:19:24.881 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:19:24.942 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:19:25.043 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:19:25.132 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:19:25.160 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:19:25.225 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:19:25.260 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:19:25.305 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:19:25.373 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:19:25.396 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:19:25.442 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:19:25.469 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:19:25.588 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:19:25.621 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:19:25.661 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:19:25.694 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:19:25.723 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:19:25.756 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:19:26.079 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:19:26.111 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:19:26.200 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:19:26.232 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:19:26.278 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:19:26.730 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:19:26.763 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:19:26.835 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:19:26.903 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:19:26.934 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:19:29.106 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:19:29.137 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:19:29.579 INFO:teuthology.orchestra.run.smithi063.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 605 kB/s | 20 MB 00:33 2024-03-25T16:19:29.584 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:29.584 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.7 MB/s | 58 MB 00:33 2024-03-25T16:19:29.618 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:19:29.660 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:19:29.660 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:19:29.706 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:19:29.741 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:19:29.830 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:19:29.830 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:19:29.830 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:19:29.830 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:19:29.859 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:19:30.169 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:19:30.170 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:19:31.818 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:19:31.910 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:19:31.959 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:19:31.985 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:19:32.016 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:19:32.032 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:19:32.217 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:19:32.272 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:19:32.621 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:19:32.661 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:19:32.737 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:19:32.826 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:19:32.863 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:19:32.917 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:19:32.954 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:19:32.999 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:19:33.068 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:19:33.090 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:19:33.153 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:19:33.180 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:19:33.291 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:19:33.323 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:19:33.364 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:19:33.396 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:19:33.426 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:19:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:19:33.786 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:19:33.821 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:19:33.903 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:19:33.934 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:19:33.974 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:19:34.452 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:19:34.482 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:19:34.537 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:19:34.608 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:19:34.643 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:19:36.820 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:19:36.856 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:19:37.434 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:19:37.468 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:19:37.544 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:19:37.544 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:19:37.544 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:19:37.544 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:19:37.577 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:19:39.027 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:19:39.028 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /sys 2024-03-25T16:19:39.028 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /proc 2024-03-25T16:19:39.028 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /mnt 2024-03-25T16:19:39.028 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /var/tmp 2024-03-25T16:19:39.028 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /home 2024-03-25T16:19:39.028 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /root 2024-03-25T16:19:39.028 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /tmp 2024-03-25T16:19:39.028 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:19:39.927 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:19:39.951 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:19:39.951 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:19:39.951 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:19:39.951 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:19:39.951 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:19:39.951 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:19:39.952 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:19:40.038 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:19:40.038 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:19:41.179 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:19:41.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:19:41.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:19:41.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:19:41.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:19:41.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:19:41.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:19:41.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:19:41.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:19:41.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:19:41.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:19:41.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:19:41.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:19:41.181 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:19:41.181 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:19:41.181 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:19:41.181 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:19:41.181 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:19:41.181 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:19:41.181 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:19:41.181 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:19:41.181 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:19:41.181 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:19:41.181 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:19:41.182 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:19:41.182 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:19:41.182 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:19:41.182 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:19:41.182 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:19:41.182 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:19:41.182 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:19:41.182 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:19:41.182 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:19:41.182 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:19:42.151 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:19:42.151 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:19:42.151 INFO:teuthology.orchestra.run.smithi081.stdout:Upgraded: 2024-03-25T16:19:42.151 INFO:teuthology.orchestra.run.smithi081.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.151 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.151 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:19:42.151 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:19:42.151 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.151 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.151 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.151 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.151 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:19:42.152 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:19:42.152 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:19:42.152 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.152 INFO:teuthology.orchestra.run.smithi081.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:19:42.152 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:19:42.152 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:19:42.152 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.152 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:19:42.152 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.152 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:19:42.152 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:19:42.152 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:19:42.153 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.153 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.153 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.153 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:19:42.153 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.153 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.153 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:42.153 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:19:42.153 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:19:42.153 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:19:42.153 INFO:teuthology.orchestra.run.smithi081.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:19:42.153 INFO:teuthology.orchestra.run.smithi081.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:19:42.153 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:19:42.153 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:19:42.469 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-test 2024-03-25T16:19:43.023 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:00:50 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:19:43.171 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:19:43.172 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:19:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T16:19:43.172 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:19:43.172 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:19:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:19:43.172 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T16:19:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:19:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:19:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:19:43.173 INFO:teuthology.orchestra.run.smithi081.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:19:43.173 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:19:43.173 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:19:43.173 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:19:43.173 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:19:43.173 INFO:teuthology.orchestra.run.smithi081.stdout:Install 6 Packages 2024-03-25T16:19:43.173 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:19:43.174 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 48 M 2024-03-25T16:19:43.174 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 204 M 2024-03-25T16:19:43.174 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:19:43.373 INFO:teuthology.orchestra.run.smithi081.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-25T16:19:43.423 INFO:teuthology.orchestra.run.smithi081.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T16:19:43.456 INFO:teuthology.orchestra.run.smithi081.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T16:19:43.923 INFO:teuthology.orchestra.run.smithi081.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 231 kB/s | 163 kB 00:00 2024-03-25T16:19:44.040 INFO:teuthology.orchestra.run.smithi081.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 321 kB/s | 187 kB 00:00 2024-03-25T16:19:47.365 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:19:47.366 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /sys 2024-03-25T16:19:47.366 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /proc 2024-03-25T16:19:47.366 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /mnt 2024-03-25T16:19:47.366 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /var/tmp 2024-03-25T16:19:47.366 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /home 2024-03-25T16:19:47.366 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /root 2024-03-25T16:19:47.366 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /tmp 2024-03-25T16:19:47.366 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:19:48.304 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:19:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:19:48.328 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:19:48.328 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:19:48.328 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:19:48.328 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:19:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:19:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:19:48.406 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:19:48.406 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:19:49.511 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:19:49.511 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:19:49.511 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:19:49.511 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:19:49.511 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:19:49.511 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:19:49.511 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:19:49.511 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:19:49.511 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:19:49.511 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:19:49.512 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:19:49.512 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:19:49.512 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:19:49.512 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:19:49.512 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:19:49.512 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:19:49.512 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:19:49.512 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:19:49.512 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:19:49.512 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:19:49.512 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:19:49.512 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:19:49.513 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:19:49.513 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:19:49.513 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:19:49.513 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:19:49.513 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:19:49.513 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:19:49.513 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:19:49.513 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:19:49.514 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:19:49.515 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:19:49.516 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:19:49.516 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:19:50.567 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:19:50.567 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:19:50.567 INFO:teuthology.orchestra.run.smithi063.stdout:Upgraded: 2024-03-25T16:19:50.567 INFO:teuthology.orchestra.run.smithi063.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:50.567 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:50.567 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:19:50.567 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:19:50.568 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:50.568 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:50.568 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:50.568 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:50.568 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:19:50.568 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:19:50.568 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:19:50.568 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:50.568 INFO:teuthology.orchestra.run.smithi063.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:19:50.568 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:19:50.568 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:19:50.568 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:50.568 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:19:50.569 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:50.569 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:19:50.569 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:19:50.569 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:19:50.569 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:50.569 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:50.569 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:50.569 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:19:50.569 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:50.569 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:50.569 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:50.569 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:19:50.569 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:19:50.570 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:19:50.570 INFO:teuthology.orchestra.run.smithi063.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:19:50.570 INFO:teuthology.orchestra.run.smithi063.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:19:50.570 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:19:50.570 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:19:50.920 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-test 2024-03-25T16:19:51.512 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:58 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:19:51.665 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:19:51.666 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:19:51.666 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T16:19:51.666 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:19:51.666 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:19:51.666 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:19:51.666 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T16:19:51.666 INFO:teuthology.orchestra.run.smithi063.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:19:51.667 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:19:51.667 INFO:teuthology.orchestra.run.smithi063.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:19:51.667 INFO:teuthology.orchestra.run.smithi063.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:19:51.667 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:19:51.667 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:19:51.667 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:19:51.667 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:19:51.667 INFO:teuthology.orchestra.run.smithi063.stdout:Install 6 Packages 2024-03-25T16:19:51.667 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:19:51.668 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 48 M 2024-03-25T16:19:51.668 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 204 M 2024-03-25T16:19:51.668 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:19:51.888 INFO:teuthology.orchestra.run.smithi063.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T16:19:51.946 INFO:teuthology.orchestra.run.smithi063.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-25T16:19:51.980 INFO:teuthology.orchestra.run.smithi063.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T16:19:52.139 INFO:teuthology.orchestra.run.smithi063.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-25T16:19:52.447 INFO:teuthology.orchestra.run.smithi063.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 222 kB/s | 163 kB 00:00 2024-03-25T16:20:19.479 INFO:teuthology.orchestra.run.smithi081.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.3 MB/s | 47 MB 00:36 2024-03-25T16:20:19.481 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:19.482 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.3 MB/s | 48 MB 00:36 2024-03-25T16:20:19.494 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:20:19.513 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:20:19.513 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:20:19.890 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:20:19.891 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:20:20.467 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:20:20.537 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:20:20.576 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:20:20.618 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:20:20.667 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:20:20.691 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:20:25.163 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:20:25.239 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:20:27.624 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:20:27.624 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:20:27.625 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:20:27.625 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:20:27.625 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:20:27.625 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:20:27.982 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:20:27.982 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:20:27.982 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:20:27.983 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:27.983 INFO:teuthology.orchestra.run.smithi081.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:20:27.983 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:27.983 INFO:teuthology.orchestra.run.smithi081.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:20:27.983 INFO:teuthology.orchestra.run.smithi081.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:20:27.983 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:20:27.983 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:20:27.983 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:20:28.316 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph 2024-03-25T16:20:28.879 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:35 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:20:29.038 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:20:29.041 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================= 2024-03-25T16:20:29.041 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T16:20:29.041 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================= 2024-03-25T16:20:29.042 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:20:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:20:29.042 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T16:20:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:20:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:20:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:20:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:20:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:20:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:20:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:20:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:20:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:20:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:20:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:20:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:20:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:20:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:20:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:20:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:20:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:20:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:20:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:20:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:20:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:20:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:20:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:20:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:20:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:20:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:20:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:20:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:20:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:20:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:20:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:20:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:20:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:20:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:20:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:20:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:20:29.048 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:20:29.048 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================= 2024-03-25T16:20:29.049 INFO:teuthology.orchestra.run.smithi081.stdout:Install 36 Packages 2024-03-25T16:20:29.049 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:20:29.050 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 30 M 2024-03-25T16:20:29.050 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 107 M 2024-03-25T16:20:29.050 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:20:29.957 INFO:teuthology.orchestra.run.smithi081.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 20 kB/s | 6.6 kB 00:00 2024-03-25T16:20:32.099 INFO:teuthology.orchestra.run.smithi081.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 617 kB/s | 1.5 MB 00:02 2024-03-25T16:20:34.108 INFO:teuthology.orchestra.run.smithi081.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 485 kB/s | 2.1 MB 00:04 2024-03-25T16:20:34.600 INFO:teuthology.orchestra.run.smithi081.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 504 kB/s | 247 kB 00:00 2024-03-25T16:20:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T16:20:34.758 INFO:teuthology.orchestra.run.smithi081.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:20:34.792 INFO:teuthology.orchestra.run.smithi081.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 942 kB/s | 32 kB 00:00 2024-03-25T16:20:34.826 INFO:teuthology.orchestra.run.smithi081.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:20:34.868 INFO:teuthology.orchestra.run.smithi081.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 928 kB/s | 39 kB 00:00 2024-03-25T16:20:34.926 INFO:teuthology.orchestra.run.smithi081.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-25T16:20:35.026 INFO:teuthology.orchestra.run.smithi081.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-25T16:20:35.077 INFO:teuthology.orchestra.run.smithi081.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-25T16:20:35.127 INFO:teuthology.orchestra.run.smithi081.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T16:20:35.169 INFO:teuthology.orchestra.run.smithi081.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:20:35.336 INFO:teuthology.orchestra.run.smithi081.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 750 kB/s | 125 kB 00:00 2024-03-25T16:20:35.411 INFO:teuthology.orchestra.run.smithi081.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 298 kB/s | 22 kB 00:00 2024-03-25T16:20:35.561 INFO:teuthology.orchestra.run.smithi081.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-25T16:20:35.612 INFO:teuthology.orchestra.run.smithi081.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 869 kB/s | 43 kB 00:00 2024-03-25T16:20:35.670 INFO:teuthology.orchestra.run.smithi081.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-03-25T16:20:35.737 INFO:teuthology.orchestra.run.smithi081.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.3 MB/s | 358 kB 00:00 2024-03-25T16:20:35.779 INFO:teuthology.orchestra.run.smithi081.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2024-03-25T16:20:35.821 INFO:teuthology.orchestra.run.smithi081.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 424 kB/s | 18 kB 00:00 2024-03-25T16:20:35.863 INFO:teuthology.orchestra.run.smithi081.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 557 kB/s | 23 kB 00:00 2024-03-25T16:20:35.905 INFO:teuthology.orchestra.run.smithi081.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 468 kB/s | 19 kB 00:00 2024-03-25T16:20:35.956 INFO:teuthology.orchestra.run.smithi081.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 394 kB/s | 20 kB 00:00 2024-03-25T16:20:36.006 INFO:teuthology.orchestra.run.smithi081.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 929 kB/s | 46 kB 00:00 2024-03-25T16:20:36.048 INFO:teuthology.orchestra.run.smithi081.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-03-25T16:20:36.106 INFO:teuthology.orchestra.run.smithi081.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2024-03-25T16:20:36.148 INFO:teuthology.orchestra.run.smithi081.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-03-25T16:20:36.191 INFO:teuthology.orchestra.run.smithi081.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-03-25T16:20:36.232 INFO:teuthology.orchestra.run.smithi081.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 857 kB/s | 36 kB 00:00 2024-03-25T16:20:36.299 INFO:teuthology.orchestra.run.smithi081.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-25T16:20:36.358 INFO:teuthology.orchestra.run.smithi081.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.2 MB/s | 427 kB 00:00 2024-03-25T16:20:36.433 INFO:teuthology.orchestra.run.smithi081.stdout:(34/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 741 kB/s | 4.7 MB 00:06 2024-03-25T16:20:36.458 INFO:teuthology.orchestra.run.smithi081.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-25T16:20:58.298 INFO:teuthology.orchestra.run.smithi081.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 656 kB/s | 17 MB 00:26 2024-03-25T16:20:58.301 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:58.302 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.0 MB/s | 30 MB 00:29 2024-03-25T16:20:58.324 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:20:58.366 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:20:58.366 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:20:58.759 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:20:58.759 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:21:00.127 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:21:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:21:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:21:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:21:00.331 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:21:00.376 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:21:00.409 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:21:00.500 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:21:00.587 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:21:00.628 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:21:00.700 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:21:00.754 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:21:00.792 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:21:00.841 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:21:00.896 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:21:01.080 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:21:01.130 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:21:01.169 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:21:01.203 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:21:01.234 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:21:01.310 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:21:01.353 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:21:01.386 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:21:01.426 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:21:01.461 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:21:01.496 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:21:01.584 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:21:01.627 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:21:01.699 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:21:01.835 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:21:01.960 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:21:02.119 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:21:02.149 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:21:02.173 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:21:02.173 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:02.174 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:21:02.174 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:21:02.174 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:21:02.174 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:03.667 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:21:03.695 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:21:03.695 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:03.695 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:21:03.695 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:21:03.696 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:21:03.696 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:04.134 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:21:04.157 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:21:04.158 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:04.158 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:21:04.158 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:21:04.158 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:21:04.158 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:04.360 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:21:04.386 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:21:04.386 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:04.387 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:21:04.387 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:21:04.387 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:21:04.387 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:04.449 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:21:05.910 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:21:05.910 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:21:05.910 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:21:05.910 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:21:05.910 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:21:05.910 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:21:05.910 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:21:05.911 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:21:05.911 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:21:05.911 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:21:05.911 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:21:05.911 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:21:05.911 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:21:05.911 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:21:05.911 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:21:05.911 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:21:05.911 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:21:05.911 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:21:05.911 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:21:05.911 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:21:05.912 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:21:05.912 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:21:05.912 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:21:05.912 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:21:05.912 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:21:05.912 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:21:05.912 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:21:05.912 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:21:05.912 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:21:05.913 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:21:05.913 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:21:05.913 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:21:05.913 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:21:05.913 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:21:05.913 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:21:05.914 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:21:07.002 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:21:07.002 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:07.002 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:21:07.003 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:07.003 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:07.003 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:07.003 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:07.003 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:07.003 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:07.003 INFO:teuthology.orchestra.run.smithi081.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:21:07.003 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:21:07.003 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:21:07.003 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:21:07.003 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:21:07.003 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:21:07.003 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:21:07.004 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:21:07.004 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:21:07.004 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:21:07.004 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:21:07.004 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:21:07.004 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:21:07.004 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:21:07.004 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:21:07.004 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:21:07.004 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:21:07.004 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:21:07.004 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:21:07.005 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:21:07.005 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:21:07.005 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:21:07.005 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:21:07.005 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:21:07.005 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:21:07.005 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:21:07.005 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:21:07.005 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:21:07.005 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:21:07.005 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:21:07.005 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:07.005 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:21:07.350 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-base 2024-03-25T16:21:07.914 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:14 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:21:08.012 INFO:teuthology.orchestra.run.smithi081.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:21:08.064 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:21:08.065 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T16:21:08.065 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:21:08.126 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install cephadm 2024-03-25T16:21:08.682 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:15 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:21:08.832 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:21:08.832 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:08.832 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T16:21:08.832 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:08.832 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:21:08.833 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:21:08.833 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:08.833 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:21:08.833 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:08.833 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T16:21:08.833 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:08.833 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 747 k 2024-03-25T16:21:08.834 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 754 k 2024-03-25T16:21:08.834 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:21:10.444 INFO:teuthology.orchestra.run.smithi081.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 464 kB/s | 747 kB 00:01 2024-03-25T16:21:10.445 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:10.445 INFO:teuthology.orchestra.run.smithi081.stdout:Total 463 kB/s | 747 kB 00:01 2024-03-25T16:21:10.446 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:21:10.452 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:21:10.452 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:21:10.458 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:21:10.459 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:21:10.538 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:21:10.948 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:21:11.041 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:21:11.479 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:21:11.744 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:21:11.744 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:11.744 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:21:11.744 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:11.744 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:11.744 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:21:11.957 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:21:12.534 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:19 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:21:12.684 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:21:12.685 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:12.685 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T16:21:12.685 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:12.685 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:21:12.685 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:21:12.686 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:12.686 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:21:12.686 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:12.686 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T16:21:12.686 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:12.686 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 145 k 2024-03-25T16:21:12.686 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 436 k 2024-03-25T16:21:12.686 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:21:13.267 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 250 kB/s | 145 kB 00:00 2024-03-25T16:21:13.268 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:13.268 INFO:teuthology.orchestra.run.smithi081.stdout:Total 249 kB/s | 145 kB 00:00 2024-03-25T16:21:13.268 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:21:13.276 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:21:13.276 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:21:13.328 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:21:13.328 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:21:13.471 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:21:13.535 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:21:13.558 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:21:13.558 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:13.559 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:21:13.559 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:14.510 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:21:14.511 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:14.511 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:21:14.511 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:14.511 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:14.511 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:21:14.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr 2024-03-25T16:21:15.334 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:22 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:21:15.434 INFO:teuthology.orchestra.run.smithi081.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:21:15.485 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:21:15.486 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T16:21:15.486 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:21:15.550 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:21:16.109 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:23 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:21:16.258 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:21:16.260 INFO:teuthology.orchestra.run.smithi081.stdout:====================================================================================== 2024-03-25T16:21:16.260 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T16:21:16.260 INFO:teuthology.orchestra.run.smithi081.stdout:====================================================================================== 2024-03-25T16:21:16.260 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:21:16.260 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:21:16.260 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T16:21:16.260 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:21:16.260 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:21:16.260 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:21:16.260 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:21:16.261 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:21:16.261 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:21:16.261 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:21:16.261 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:21:16.261 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:21:16.261 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:21:16.261 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:21:16.261 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:16.261 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:21:16.261 INFO:teuthology.orchestra.run.smithi081.stdout:====================================================================================== 2024-03-25T16:21:16.261 INFO:teuthology.orchestra.run.smithi081.stdout:Install 12 Packages 2024-03-25T16:21:16.261 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:16.262 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 6.9 M 2024-03-25T16:21:16.262 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 26 M 2024-03-25T16:21:16.262 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:21:17.046 INFO:teuthology.orchestra.run.smithi081.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 29 kB/s | 15 kB 00:00 2024-03-25T16:21:17.188 INFO:teuthology.orchestra.run.smithi081.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-03-25T16:21:17.230 INFO:teuthology.orchestra.run.smithi081.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 5.6 MB/s | 238 kB 00:00 2024-03-25T16:21:17.338 INFO:teuthology.orchestra.run.smithi081.stdout:(4/12): ceph-grafana-dashboards-19.0.0-1439.g5d 38 kB/s | 31 kB 00:00 2024-03-25T16:21:17.597 INFO:teuthology.orchestra.run.smithi081.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.3 MB/s | 863 kB 00:00 2024-03-25T16:21:17.864 INFO:teuthology.orchestra.run.smithi081.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-03-25T16:21:18.023 INFO:teuthology.orchestra.run.smithi081.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 801 kB/s | 548 kB 00:00 2024-03-25T16:21:18.448 INFO:teuthology.orchestra.run.smithi081.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 340 kB/s | 144 kB 00:00 2024-03-25T16:21:18.665 INFO:teuthology.orchestra.run.smithi081.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 2.5 MB/s | 2.0 MB 00:00 2024-03-25T16:21:18.715 INFO:teuthology.orchestra.run.smithi081.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 115 kB/s | 31 kB 00:00 2024-03-25T16:21:19.090 INFO:teuthology.orchestra.run.smithi081.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 444 kB/s | 188 kB 00:00 2024-03-25T16:21:20.607 INFO:teuthology.orchestra.run.smithi081.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 446 kB/s | 1.8 MB 00:04 2024-03-25T16:21:20.612 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:20.612 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.6 MB/s | 6.9 MB 00:04 2024-03-25T16:21:20.631 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:21:20.651 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:21:20.651 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:21:20.829 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:21:20.829 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:21:21.364 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:21:21.441 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:21:21.484 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:21:21.532 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:21:21.570 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:21:21.667 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:21:21.802 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:21:21.889 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:21:21.933 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:21:21.961 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:21:21.993 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:21:22.235 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:21:22.270 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:21:23.060 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:21:23.060 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:21:23.060 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:21:23.060 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:21:23.060 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:21:23.060 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:21:23.060 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:21:23.061 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:21:23.061 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:21:23.061 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:21:23.061 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:21:23.061 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:21:23.409 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:21:23.409 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:23.410 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:21:23.410 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:21:23.410 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:23.410 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:23.410 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:23.410 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:21:23.410 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:21:23.410 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:21:23.410 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:21:23.410 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:21:23.410 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:21:23.410 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:21:23.411 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:21:23.411 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:23.411 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:21:23.668 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:21:24.231 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:31 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:21:24.382 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:21:24.383 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================================ 2024-03-25T16:21:24.383 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T16:21:24.384 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================================ 2024-03-25T16:21:24.384 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:21:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:21:24.384 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T16:21:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:21:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:21:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:21:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:21:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:21:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:21:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:21:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:21:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:21:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:21:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:21:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:24.385 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:21:24.385 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================================ 2024-03-25T16:21:24.385 INFO:teuthology.orchestra.run.smithi081.stdout:Install 12 Packages 2024-03-25T16:21:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:24.386 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 37 M 2024-03-25T16:21:24.386 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 209 M 2024-03-25T16:21:24.386 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:21:24.714 INFO:teuthology.orchestra.run.smithi081.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 142 kB/s | 32 kB 00:00 2024-03-25T16:21:24.773 INFO:teuthology.orchestra.run.smithi081.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 298 kB/s | 17 kB 00:00 2024-03-25T16:21:24.815 INFO:teuthology.orchestra.run.smithi081.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 821 kB/s | 34 kB 00:00 2024-03-25T16:21:24.915 INFO:teuthology.orchestra.run.smithi081.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.1 MB/s | 3.0 MB 00:00 2024-03-25T16:21:25.274 INFO:teuthology.orchestra.run.smithi081.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 10 MB/s | 4.7 MB 00:00 2024-03-25T16:21:25.340 INFO:teuthology.orchestra.run.smithi081.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 4.8 MB 00:00 2024-03-25T16:21:25.366 INFO:teuthology.orchestra.run.smithi081.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T16:21:25.816 INFO:teuthology.orchestra.run.smithi081.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-03-25T16:21:26.067 INFO:teuthology.orchestra.run.smithi081.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-25T16:21:26.125 INFO:teuthology.orchestra.run.smithi081.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 633 kB/s | 195 kB 00:00 2024-03-25T16:21:26.267 INFO:teuthology.orchestra.run.smithi081.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-25T16:21:26.313 INFO:teuthology.orchestra.run.smithi063.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 512 kB/s | 47 MB 01:34 2024-03-25T16:21:26.315 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:26.316 INFO:teuthology.orchestra.run.smithi063.stdout:Total 521 kB/s | 48 MB 01:34 2024-03-25T16:21:26.327 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:21:26.348 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:21:26.348 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:21:26.738 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:21:26.742 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:21:27.346 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:21:27.406 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:21:27.452 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:21:27.492 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:21:27.531 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:21:27.557 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:21:29.475 INFO:teuthology.orchestra.run.smithi081.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.5 MB/s | 7.4 MB 00:04 2024-03-25T16:21:29.479 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:29.480 INFO:teuthology.orchestra.run.smithi081.stdout:Total 7.3 MB/s | 37 MB 00:05 2024-03-25T16:21:29.523 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:21:29.549 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:21:29.549 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:21:29.913 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:21:29.913 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:21:30.641 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:21:30.723 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:21:30.754 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:21:30.846 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:21:30.882 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:21:31.058 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:21:31.090 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:21:31.169 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:21:31.829 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:21:31.935 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:21:32.290 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:21:32.384 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:21:33.168 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:21:34.216 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:21:34.239 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:21:34.691 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:21:34.691 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:21:34.692 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:21:34.692 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:21:34.692 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:21:34.692 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:21:35.107 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:21:35.107 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:21:35.107 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:21:35.107 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:35.107 INFO:teuthology.orchestra.run.smithi063.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:21:35.107 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:35.107 INFO:teuthology.orchestra.run.smithi063.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:21:35.107 INFO:teuthology.orchestra.run.smithi063.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:21:35.107 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:21:35.107 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:21:35.107 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:21:35.413 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph 2024-03-25T16:21:35.987 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:42 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:21:36.155 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:21:36.158 INFO:teuthology.orchestra.run.smithi063.stdout:========================================================================================= 2024-03-25T16:21:36.158 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T16:21:36.158 INFO:teuthology.orchestra.run.smithi063.stdout:========================================================================================= 2024-03-25T16:21:36.158 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:21:36.158 INFO:teuthology.orchestra.run.smithi063.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:21:36.158 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T16:21:36.159 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:21:36.159 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:21:36.159 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:21:36.159 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:21:36.159 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:21:36.159 INFO:teuthology.orchestra.run.smithi063.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:21:36.159 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:21:36.159 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:21:36.159 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:21:36.159 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:21:36.159 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:21:36.159 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:21:36.159 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:21:36.160 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:21:36.160 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:21:36.160 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:21:36.160 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:21:36.160 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:21:36.160 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:21:36.160 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:21:36.160 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:21:36.160 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:21:36.160 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:21:36.160 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:21:36.160 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:21:36.160 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:21:36.161 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:21:36.161 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:21:36.161 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:21:36.161 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:21:36.161 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:21:36.161 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:21:36.161 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:21:36.161 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:21:36.161 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:21:36.161 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:21:36.161 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:21:36.161 INFO:teuthology.orchestra.run.smithi063.stdout:========================================================================================= 2024-03-25T16:21:36.161 INFO:teuthology.orchestra.run.smithi063.stdout:Install 36 Packages 2024-03-25T16:21:36.161 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:21:36.162 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 30 M 2024-03-25T16:21:36.162 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 107 M 2024-03-25T16:21:36.162 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:21:36.903 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:21:36.903 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:21:36.903 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:21:36.903 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:21:36.903 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:21:36.903 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:21:36.903 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:21:36.903 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:21:36.903 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:21:36.903 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:21:36.904 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:21:36.904 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:21:36.957 INFO:teuthology.orchestra.run.smithi063.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 13 kB/s | 6.6 kB 00:00 2024-03-25T16:21:37.361 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:21:37.361 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:37.361 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:21:37.362 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:37.362 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:21:37.362 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:21:37.362 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:21:37.362 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:21:37.362 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:21:37.362 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:21:37.362 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:21:37.362 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:21:37.362 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:21:37.362 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:21:37.362 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:21:37.362 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:37.362 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:21:37.721 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:21:38.290 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:45 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:21:38.439 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:21:38.441 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T16:21:38.441 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T16:21:38.441 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T16:21:38.441 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:21:38.441 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:21:38.441 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T16:21:38.441 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:21:38.441 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:21:38.441 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:21:38.441 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:21:38.441 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:21:38.442 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:21:38.442 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:21:38.442 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:21:38.442 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:21:38.442 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:21:38.442 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:21:38.442 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:21:38.442 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:38.442 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:21:38.442 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T16:21:38.442 INFO:teuthology.orchestra.run.smithi081.stdout:Install 13 Packages 2024-03-25T16:21:38.442 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:38.443 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 2.0 M 2024-03-25T16:21:38.443 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 26 M 2024-03-25T16:21:38.443 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:21:38.799 INFO:teuthology.orchestra.run.smithi081.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 126 kB/s | 18 kB 00:00 2024-03-25T16:21:38.824 INFO:teuthology.orchestra.run.smithi081.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 144 kB/s | 24 kB 00:00 2024-03-25T16:21:38.874 INFO:teuthology.orchestra.run.smithi081.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-25T16:21:38.899 INFO:teuthology.orchestra.run.smithi081.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:21:38.924 INFO:teuthology.orchestra.run.smithi081.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:21:38.950 INFO:teuthology.orchestra.run.smithi081.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-03-25T16:21:39.075 INFO:teuthology.orchestra.run.smithi081.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 120 kB/s | 15 kB 00:00 2024-03-25T16:21:39.100 INFO:teuthology.orchestra.run.smithi081.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 183 kB/s | 32 kB 00:00 2024-03-25T16:21:39.217 INFO:teuthology.orchestra.run.smithi081.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 93 kB/s | 52 kB 00:00 2024-03-25T16:21:39.242 INFO:teuthology.orchestra.run.smithi081.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.2 MB/s | 203 kB 00:00 2024-03-25T16:21:39.309 INFO:teuthology.orchestra.run.smithi081.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-25T16:21:39.368 INFO:teuthology.orchestra.run.smithi081.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.8 MB/s | 1.0 MB 00:00 2024-03-25T16:21:39.393 INFO:teuthology.orchestra.run.smithi081.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 337 kB/s | 59 kB 00:00 2024-03-25T16:21:39.395 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:39.395 INFO:teuthology.orchestra.run.smithi081.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-25T16:21:39.396 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:21:39.408 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:21:39.408 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:21:39.423 INFO:teuthology.orchestra.run.smithi063.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 730 kB/s | 2.1 MB 00:02 2024-03-25T16:21:39.497 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:21:39.501 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:21:40.033 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:21:40.197 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:21:40.232 INFO:teuthology.orchestra.run.smithi063.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 403 kB/s | 1.5 MB 00:03 2024-03-25T16:21:40.260 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:21:40.315 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:21:40.346 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:21:40.387 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:21:40.480 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:21:40.598 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:21:40.654 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:21:40.673 INFO:teuthology.orchestra.run.smithi063.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 560 kB/s | 247 kB 00:00 2024-03-25T16:21:40.790 INFO:teuthology.orchestra.run.smithi063.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T16:21:40.832 INFO:teuthology.orchestra.run.smithi063.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:21:40.866 INFO:teuthology.orchestra.run.smithi063.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2024-03-25T16:21:40.900 INFO:teuthology.orchestra.run.smithi063.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:21:40.933 INFO:teuthology.orchestra.run.smithi063.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:21:40.975 INFO:teuthology.orchestra.run.smithi063.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:21:41.050 INFO:teuthology.orchestra.run.smithi063.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T16:21:41.076 INFO:teuthology.orchestra.run.smithi063.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 4.0 MB/s | 103 kB 00:00 2024-03-25T16:21:41.118 INFO:teuthology.orchestra.run.smithi063.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T16:21:41.159 INFO:teuthology.orchestra.run.smithi063.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:21:41.371 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:21:41.418 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:21:41.452 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:21:41.500 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:21:41.522 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:21:41.585 INFO:teuthology.orchestra.run.smithi063.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 294 kB/s | 125 kB 00:00 2024-03-25T16:21:41.669 INFO:teuthology.orchestra.run.smithi063.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-03-25T16:21:41.877 INFO:teuthology.orchestra.run.smithi063.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 725 kB/s | 151 kB 00:00 2024-03-25T16:21:41.927 INFO:teuthology.orchestra.run.smithi063.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 872 kB/s | 43 kB 00:00 2024-03-25T16:21:42.011 INFO:teuthology.orchestra.run.smithi063.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-03-25T16:21:42.112 INFO:teuthology.orchestra.run.smithi063.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-03-25T16:21:42.170 INFO:teuthology.orchestra.run.smithi063.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2024-03-25T16:21:42.228 INFO:teuthology.orchestra.run.smithi063.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-03-25T16:21:42.287 INFO:teuthology.orchestra.run.smithi063.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 398 kB/s | 23 kB 00:00 2024-03-25T16:21:42.354 INFO:teuthology.orchestra.run.smithi063.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2024-03-25T16:21:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:21:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:21:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:21:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:21:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:21:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:21:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:21:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:21:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:21:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:21:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:21:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:21:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:21:42.413 INFO:teuthology.orchestra.run.smithi063.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 338 kB/s | 20 kB 00:00 2024-03-25T16:21:42.471 INFO:teuthology.orchestra.run.smithi063.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 797 kB/s | 46 kB 00:00 2024-03-25T16:21:42.522 INFO:teuthology.orchestra.run.smithi063.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-25T16:21:42.588 INFO:teuthology.orchestra.run.smithi063.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-25T16:21:42.647 INFO:teuthology.orchestra.run.smithi063.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-25T16:21:42.706 INFO:teuthology.orchestra.run.smithi063.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-25T16:21:42.716 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:21:42.716 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:42.716 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:21:42.716 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:42.716 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:21:42.716 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:21:42.716 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:21:42.717 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:21:42.717 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:21:42.717 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:21:42.717 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:21:42.717 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:21:42.717 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:21:42.717 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:21:42.717 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:21:42.717 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:21:42.717 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:42.717 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:21:42.764 INFO:teuthology.orchestra.run.smithi063.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 615 kB/s | 36 kB 00:00 2024-03-25T16:21:42.831 INFO:teuthology.orchestra.run.smithi063.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-25T16:21:42.906 INFO:teuthology.orchestra.run.smithi063.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-03-25T16:21:42.965 INFO:teuthology.orchestra.run.smithi063.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 343 kB/s | 20 kB 00:00 2024-03-25T16:21:42.968 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:21:43.529 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:50 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:21:43.678 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:21:43.679 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T16:21:43.679 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T16:21:43.679 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T16:21:43.679 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:21:43.679 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:21:43.679 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T16:21:43.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:21:43.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:21:43.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:21:43.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:21:43.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:21:43.680 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:43.680 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:21:43.680 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T16:21:43.680 INFO:teuthology.orchestra.run.smithi081.stdout:Install 6 Packages 2024-03-25T16:21:43.680 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:43.681 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 6.8 M 2024-03-25T16:21:43.681 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 33 M 2024-03-25T16:21:43.681 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:21:44.137 INFO:teuthology.orchestra.run.smithi081.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-25T16:21:44.420 INFO:teuthology.orchestra.run.smithi081.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.9 MB/s | 548 kB 00:00 2024-03-25T16:21:44.446 INFO:teuthology.orchestra.run.smithi081.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 239 kB/s | 138 kB 00:00 2024-03-25T16:21:44.473 INFO:teuthology.orchestra.run.smithi081.stdout:(4/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 410 kB/s | 248 kB 00:00 2024-03-25T16:21:44.515 INFO:teuthology.orchestra.run.smithi081.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 612 kB/s | 58 kB 00:00 2024-03-25T16:21:44.540 INFO:teuthology.orchestra.run.smithi081.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 541 kB/s | 51 kB 00:00 2024-03-25T16:21:44.543 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:44.543 INFO:teuthology.orchestra.run.smithi081.stdout:Total 7.9 MB/s | 6.8 MB 00:00 2024-03-25T16:21:44.556 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:21:44.556 INFO:teuthology.orchestra.run.smithi063.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 631 kB/s | 4.7 MB 00:07 2024-03-25T16:21:44.566 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:21:44.567 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:21:44.655 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:21:44.656 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:21:44.971 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:21:45.153 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:21:45.201 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:21:45.709 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:21:45.813 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:21:45.873 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:21:45.896 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:21:46.717 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:21:46.717 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:21:46.717 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:21:46.718 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:21:46.718 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:21:46.718 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:21:47.005 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:21:47.005 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:47.005 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:21:47.005 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:47.005 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:21:47.005 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:21:47.005 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:21:47.006 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:21:47.006 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:21:47.006 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:47.006 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:21:47.253 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-fuse 2024-03-25T16:21:47.814 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:54 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:21:47.963 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:21:47.964 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:47.964 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T16:21:47.964 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:47.964 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:21:47.964 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:21:47.964 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T16:21:47.964 INFO:teuthology.orchestra.run.smithi081.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:21:47.964 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:47.964 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:21:47.964 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:47.964 INFO:teuthology.orchestra.run.smithi081.stdout:Install 2 Packages 2024-03-25T16:21:47.965 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:47.965 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 964 k 2024-03-25T16:21:47.965 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 2.8 M 2024-03-25T16:21:47.965 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:21:48.075 INFO:teuthology.orchestra.run.smithi081.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 717 kB/s | 79 kB 00:00 2024-03-25T16:21:49.642 INFO:teuthology.orchestra.run.smithi081.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 528 kB/s | 885 kB 00:01 2024-03-25T16:21:49.643 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:49.644 INFO:teuthology.orchestra.run.smithi081.stdout:Total 574 kB/s | 964 kB 00:01 2024-03-25T16:21:49.644 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:21:49.666 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:21:49.667 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:21:49.743 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:21:49.744 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:21:49.925 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:21:50.075 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:21:50.141 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:21:50.848 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:21:50.848 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:21:51.147 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:21:51.148 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:51.148 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:21:51.148 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:21:51.148 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:51.148 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:21:51.454 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-volume 2024-03-25T16:21:52.028 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:59 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:21:52.180 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:21:52.181 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:52.181 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T16:21:52.181 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:52.181 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:21:52.181 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:21:52.181 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T16:21:52.181 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:21:52.181 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:52.181 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:21:52.182 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:52.182 INFO:teuthology.orchestra.run.smithi081.stdout:Install 2 Packages 2024-03-25T16:21:52.182 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:52.182 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 332 k 2024-03-25T16:21:52.182 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 1.5 M 2024-03-25T16:21:52.182 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:21:52.296 INFO:teuthology.orchestra.run.smithi081.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 603 kB/s | 69 kB 00:00 2024-03-25T16:21:53.163 INFO:teuthology.orchestra.run.smithi081.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 269 kB/s | 264 kB 00:00 2024-03-25T16:21:53.165 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:53.165 INFO:teuthology.orchestra.run.smithi081.stdout:Total 338 kB/s | 332 kB 00:00 2024-03-25T16:21:53.165 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:21:53.178 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:21:53.178 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:21:53.215 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:21:53.216 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:21:53.364 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:21:53.545 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:21:53.586 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:21:53.599 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:21:53.599 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:53.599 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:21:53.599 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:54.341 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:21:54.665 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:21:54.665 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:54.666 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:21:54.666 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:54.666 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:21:54.666 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:54.666 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:21:55.045 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librados-devel 2024-03-25T16:21:55.637 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:03:02 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:21:55.786 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:21:55.787 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:55.787 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T16:21:55.787 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:55.787 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:21:55.788 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:21:55.788 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:55.788 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:21:55.788 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:55.788 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T16:21:55.788 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:55.788 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 127 k 2024-03-25T16:21:55.788 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 456 k 2024-03-25T16:21:55.788 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:21:56.486 INFO:teuthology.orchestra.run.smithi081.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 181 kB/s | 127 kB 00:00 2024-03-25T16:21:56.487 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:56.487 INFO:teuthology.orchestra.run.smithi081.stdout:Total 181 kB/s | 127 kB 00:00 2024-03-25T16:21:56.487 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:21:56.493 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:21:56.493 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:21:56.551 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:21:56.551 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:21:56.701 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:21:56.801 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:57.275 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:57.563 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:21:57.563 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:57.563 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:21:57.563 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:57.564 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:57.564 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:21:57.793 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install libcephfs2 2024-03-25T16:21:58.367 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:03:05 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:21:58.466 INFO:teuthology.orchestra.run.smithi081.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:21:58.520 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:21:58.521 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T16:21:58.521 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:21:58.590 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install libcephfs-devel 2024-03-25T16:21:59.151 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:03:06 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:21:59.299 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:21:59.299 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:59.299 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T16:21:59.299 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:59.300 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:21:59.300 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:21:59.300 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:59.300 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:21:59.300 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:21:59.300 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T16:21:59.300 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:21:59.300 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 31 k 2024-03-25T16:21:59.300 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 140 k 2024-03-25T16:21:59.300 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:21:59.722 INFO:teuthology.orchestra.run.smithi081.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 74 kB/s | 31 kB 00:00 2024-03-25T16:21:59.723 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:59.723 INFO:teuthology.orchestra.run.smithi081.stdout:Total 74 kB/s | 31 kB 00:00 2024-03-25T16:21:59.723 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:21:59.727 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:21:59.730 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:21:59.756 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:21:59.756 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:21:59.886 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:22:00.004 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:00.395 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:00.646 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:00.646 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:00.646 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:22:00.646 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:00.646 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:00.646 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:22:00.819 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librados2 2024-03-25T16:22:01.386 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:03:08 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:01.484 INFO:teuthology.orchestra.run.smithi081.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:01.535 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:22:01.536 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T16:22:01.536 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:22:01.579 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librbd1 2024-03-25T16:22:02.145 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:03:09 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:02.242 INFO:teuthology.orchestra.run.smithi081.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:02.294 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:22:02.295 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T16:22:02.295 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:22:02.341 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rados 2024-03-25T16:22:02.563 INFO:teuthology.orchestra.run.smithi063.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 742 kB/s | 17 MB 00:23 2024-03-25T16:22:02.567 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:02.567 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.1 MB/s | 30 MB 00:26 2024-03-25T16:22:02.593 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:22:02.639 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:22:02.639 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:22:02.902 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:03:09 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:03.000 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:03.051 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:22:03.052 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T16:22:03.052 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:22:03.055 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:22:03.055 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:22:03.096 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rgw 2024-03-25T16:22:03.659 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:03:10 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:03.754 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:03.805 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:22:03.806 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T16:22:03.806 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:22:03.851 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-cephfs 2024-03-25T16:22:04.385 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:22:04.423 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:03:11 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:04.450 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:22:04.494 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:22:04.525 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:04.526 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:22:04.580 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:22:04.581 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T16:22:04.581 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:22:04.603 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:22:04.630 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rbd 2024-03-25T16:22:04.642 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:22:04.675 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:22:04.770 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:22:04.848 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:22:04.895 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:22:04.969 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:22:05.021 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:22:05.101 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:22:05.159 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:22:05.222 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:22:05.275 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:03:12 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:05.376 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:05.417 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:22:05.429 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:22:05.430 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T16:22:05.431 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:22:05.474 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:22:05.480 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-fuse 2024-03-25T16:22:05.509 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:22:05.543 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:22:05.582 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:22:05.662 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:22:05.692 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:22:05.725 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:22:05.766 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:22:05.809 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:22:05.843 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:22:05.929 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:22:05.974 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:22:06.047 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:03:13 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:06.050 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:22:06.197 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:22:06.198 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:22:06.198 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-03-25T16:22:06.198 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:22:06.198 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:22:06.198 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:22:06.198 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:06.198 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:22:06.198 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:22:06.198 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T16:22:06.198 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:06.198 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 85 k 2024-03-25T16:22:06.198 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 231 k 2024-03-25T16:22:06.199 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:22:06.200 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:22:06.330 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:22:06.498 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:22:06.528 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:22:06.553 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:22:06.553 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:06.554 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:22:06.554 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:22:06.554 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:22:06.554 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:07.190 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 86 kB/s | 85 kB 00:00 2024-03-25T16:22:07.190 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:07.190 INFO:teuthology.orchestra.run.smithi081.stdout:Total 86 kB/s | 85 kB 00:00 2024-03-25T16:22:07.190 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:22:07.197 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:22:07.198 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:22:07.255 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:22:07.255 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:22:07.397 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:22:07.499 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:07.938 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:08.079 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:22:08.105 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:22:08.105 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:08.105 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:22:08.105 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:22:08.105 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:22:08.105 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:08.221 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:08.221 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:08.221 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:22:08.222 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:08.222 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:08.222 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:22:08.444 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-mirror 2024-03-25T16:22:08.571 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:22:08.594 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:22:08.594 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:08.595 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:22:08.595 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:22:08.595 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:22:08.595 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:08.813 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:22:08.837 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:22:08.837 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:08.837 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:22:08.837 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:22:08.837 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:22:08.837 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:22:09.010 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:03:16 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:09.164 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:22:09.165 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:22:09.165 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T16:22:09.165 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:22:09.165 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:22:09.165 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:22:09.165 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:09.165 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:22:09.166 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:22:09.166 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T16:22:09.166 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:09.166 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 3.1 M 2024-03-25T16:22:09.166 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 13 M 2024-03-25T16:22:09.166 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:22:10.535 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:22:10.535 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:22:10.535 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:22:10.535 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:22:10.535 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:22:10.535 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:22:10.535 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:22:10.536 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:22:10.536 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:22:10.536 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:22:10.536 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:22:10.536 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:22:10.536 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:22:10.536 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:22:10.536 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:22:10.536 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:22:10.536 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:22:10.536 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:22:10.536 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:22:10.536 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:22:10.537 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:22:10.537 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:22:10.537 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:22:10.537 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:22:10.537 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:22:10.537 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:22:10.538 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:22:10.538 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:22:10.538 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:22:10.538 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:22:10.538 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:22:10.538 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:22:10.539 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:22:10.539 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:22:10.539 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:22:10.539 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:22:11.492 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:22:11.493 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:11.493 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:22:11.493 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:11.493 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:11.493 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:11.493 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:11.493 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:11.493 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:11.493 INFO:teuthology.orchestra.run.smithi063.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:22:11.493 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:22:11.493 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:22:11.493 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:22:11.494 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:22:11.494 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:22:11.494 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:22:11.494 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:22:11.494 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:22:11.494 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:22:11.494 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:22:11.494 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:22:11.494 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:22:11.494 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:22:11.494 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:22:11.494 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:22:11.494 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:22:11.495 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:22:11.495 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:22:11.495 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:22:11.495 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:22:11.495 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:22:11.495 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:22:11.495 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:22:11.495 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:22:11.495 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:22:11.495 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:22:11.495 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:22:11.495 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:22:11.495 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:22:11.495 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:11.496 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:22:11.857 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-base 2024-03-25T16:22:12.434 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:03:19 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:12.535 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:12.591 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:22:12.592 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T16:22:12.592 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:22:12.668 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install cephadm 2024-03-25T16:22:13.242 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:03:20 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:13.398 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:22:13.399 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:22:13.399 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T16:22:13.399 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:22:13.399 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:22:13.399 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:22:13.399 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:13.400 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:22:13.400 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:22:13.400 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T16:22:13.400 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:13.400 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 747 k 2024-03-25T16:22:13.400 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 754 k 2024-03-25T16:22:13.400 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:22:13.979 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 657 kB/s | 3.1 MB 00:04 2024-03-25T16:22:13.979 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:13.980 INFO:teuthology.orchestra.run.smithi081.stdout:Total 657 kB/s | 3.1 MB 00:04 2024-03-25T16:22:13.980 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:22:13.991 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:22:13.991 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:22:14.068 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:22:14.069 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:22:14.498 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:22:14.563 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:14.586 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:14.587 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:14.587 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:22:14.587 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:22:14.587 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:22:14.587 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:15.126 INFO:teuthology.orchestra.run.smithi063.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 433 kB/s | 747 kB 00:01 2024-03-25T16:22:15.127 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:15.127 INFO:teuthology.orchestra.run.smithi063.stdout:Total 433 kB/s | 747 kB 00:01 2024-03-25T16:22:15.127 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:22:15.134 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:22:15.134 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:22:15.140 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:22:15.140 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:22:15.221 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:22:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:15.728 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:15.728 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:15.728 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:22:15.728 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:15.728 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:15.728 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:22:15.783 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:15.982 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-nbd 2024-03-25T16:22:16.146 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:16.454 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:16.454 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:16.454 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:22:16.455 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:16.455 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:16.455 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:22:16.557 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:03:23 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:16.648 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:22:16.710 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:22:16.711 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:22:16.711 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-03-25T16:22:16.711 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:22:16.711 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:22:16.711 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:22:16.711 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:16.711 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:22:16.711 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:22:16.711 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T16:22:16.711 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:16.711 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 171 k 2024-03-25T16:22:16.712 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 494 k 2024-03-25T16:22:16.712 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:22:17.228 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:03:24 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:17.383 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:22:17.384 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:22:17.384 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T16:22:17.384 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:22:17.384 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:22:17.385 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:22:17.385 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:17.385 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:22:17.385 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:22:17.385 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T16:22:17.385 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:17.385 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 145 k 2024-03-25T16:22:17.385 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 436 k 2024-03-25T16:22:17.385 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:22:17.783 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 160 kB/s | 171 kB 00:01 2024-03-25T16:22:17.783 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:17.783 INFO:teuthology.orchestra.run.smithi081.stdout:Total 160 kB/s | 171 kB 00:01 2024-03-25T16:22:17.784 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:22:17.793 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:22:17.793 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:22:17.854 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:22:17.855 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:22:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:22:18.097 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:18.131 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 195 kB/s | 145 kB 00:00 2024-03-25T16:22:18.132 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:18.132 INFO:teuthology.orchestra.run.smithi063.stdout:Total 194 kB/s | 145 kB 00:00 2024-03-25T16:22:18.132 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:22:18.141 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:22:18.141 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:22:18.197 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:22:18.197 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:22:18.344 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:22:18.399 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:18.423 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:18.423 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:18.423 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:22:18.423 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:18.547 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:18.785 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:18.785 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:18.785 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:22:18.785 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:18.785 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:18.785 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:22:19.015 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:22:19.453 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:19.453 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:19.453 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:22:19.453 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:19.453 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:19.453 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:22:19.581 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:03:26 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:19.693 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr 2024-03-25T16:22:19.733 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:22:19.734 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:22:19.734 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T16:22:19.734 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:22:19.734 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:22:19.734 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:22:19.734 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T16:22:19.734 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:22:19.734 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:19.734 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:22:19.734 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:22:19.734 INFO:teuthology.orchestra.run.smithi081.stdout:Install 2 Packages 2024-03-25T16:22:19.735 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:22:19.735 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 91 M 2024-03-25T16:22:19.735 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 498 M 2024-03-25T16:22:19.735 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:22:20.266 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:03:27 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:20.366 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:20.421 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:22:20.422 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T16:22:20.422 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:22:20.486 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:22:21.057 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:03:28 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:21.213 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:22:21.214 INFO:teuthology.orchestra.run.smithi063.stdout:====================================================================================== 2024-03-25T16:22:21.214 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T16:22:21.214 INFO:teuthology.orchestra.run.smithi063.stdout:====================================================================================== 2024-03-25T16:22:21.214 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:22:21.215 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:22:21.215 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T16:22:21.215 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:22:21.215 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:22:21.215 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:22:21.215 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:22:21.215 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:22:21.215 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:22:21.215 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:22:21.215 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:22:21.215 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:22:21.215 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:22:21.215 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:22:21.216 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:21.216 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:22:21.216 INFO:teuthology.orchestra.run.smithi063.stdout:====================================================================================== 2024-03-25T16:22:21.216 INFO:teuthology.orchestra.run.smithi063.stdout:Install 12 Packages 2024-03-25T16:22:21.216 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:21.216 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 6.9 M 2024-03-25T16:22:21.216 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 26 M 2024-03-25T16:22:21.217 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:22:21.784 INFO:teuthology.orchestra.run.smithi063.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 69 kB/s | 31 kB 00:00 2024-03-25T16:22:21.809 INFO:teuthology.orchestra.run.smithi063.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 32 kB/s | 15 kB 00:00 2024-03-25T16:22:21.960 INFO:teuthology.orchestra.run.smithi063.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-03-25T16:22:22.002 INFO:teuthology.orchestra.run.smithi063.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-25T16:22:22.319 INFO:teuthology.orchestra.run.smithi063.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.4 MB/s | 863 kB 00:00 2024-03-25T16:22:22.527 INFO:teuthology.orchestra.run.smithi063.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.0 MB/s | 548 kB 00:00 2024-03-25T16:22:22.553 INFO:teuthology.orchestra.run.smithi063.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 83 kB/s | 19 kB 00:00 2024-03-25T16:22:22.803 INFO:teuthology.orchestra.run.smithi063.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 7.4 MB/s | 2.0 MB 00:00 2024-03-25T16:22:22.828 INFO:teuthology.orchestra.run.smithi063.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 524 kB/s | 144 kB 00:00 2024-03-25T16:22:22.895 INFO:teuthology.orchestra.run.smithi063.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 334 kB/s | 31 kB 00:00 2024-03-25T16:22:23.020 INFO:teuthology.orchestra.run.smithi063.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 984 kB/s | 188 kB 00:00 2024-03-25T16:22:26.545 INFO:teuthology.orchestra.run.smithi063.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 350 kB/s | 1.8 MB 00:05 2024-03-25T16:22:26.549 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:26.550 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.3 MB/s | 6.9 MB 00:05 2024-03-25T16:22:26.570 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:22:26.590 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:22:26.590 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:22:26.773 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:22:26.774 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:22:27.295 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:22:27.355 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:22:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:22:27.437 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:22:27.466 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:22:27.571 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:22:27.696 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:22:27.788 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:22:27.830 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:22:27.858 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:22:27.889 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:22:28.143 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:22:28.180 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:22:28.930 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:22:28.930 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:22:28.930 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:22:28.930 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:22:28.930 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:22:28.931 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:22:28.931 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:22:28.931 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:22:28.931 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:22:28.931 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:22:28.931 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:22:28.931 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:22:29.277 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:22:29.277 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:29.277 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:22:29.277 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:22:29.277 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:29.277 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:29.278 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:29.278 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:22:29.278 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:22:29.278 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:22:29.278 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:22:29.278 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:22:29.278 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:22:29.278 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:22:29.278 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:22:29.278 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:29.278 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:22:29.547 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:22:30.113 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:03:37 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:30.270 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:22:30.272 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-03-25T16:22:30.272 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T16:22:30.272 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-03-25T16:22:30.272 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:22:30.272 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:22:30.272 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T16:22:30.272 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:22:30.272 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:22:30.272 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:22:30.272 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:22:30.272 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:22:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:22:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:22:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:22:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:22:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:22:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:22:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:30.273 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:22:30.273 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-03-25T16:22:30.273 INFO:teuthology.orchestra.run.smithi063.stdout:Install 12 Packages 2024-03-25T16:22:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:30.274 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 37 M 2024-03-25T16:22:30.274 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 209 M 2024-03-25T16:22:30.274 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:22:30.668 INFO:teuthology.orchestra.run.smithi063.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 256 kB/s | 32 kB 00:00 2024-03-25T16:22:30.718 INFO:teuthology.orchestra.run.smithi063.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 672 kB/s | 17 kB 00:00 2024-03-25T16:22:30.743 INFO:teuthology.orchestra.run.smithi063.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-25T16:22:30.902 INFO:teuthology.orchestra.run.smithi063.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-25T16:22:31.035 INFO:teuthology.orchestra.run.smithi063.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T16:22:31.069 INFO:teuthology.orchestra.run.smithi063.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 6.7 MB/s | 217 kB 00:00 2024-03-25T16:22:31.111 INFO:teuthology.orchestra.run.smithi063.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.3 MB/s | 3.0 MB 00:00 2024-03-25T16:22:31.219 INFO:teuthology.orchestra.run.smithi063.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 7.4 MB/s | 803 kB 00:00 2024-03-25T16:22:31.436 INFO:teuthology.orchestra.run.smithi063.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-25T16:22:31.461 INFO:teuthology.orchestra.run.smithi063.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 816 kB/s | 195 kB 00:00 2024-03-25T16:22:31.695 INFO:teuthology.orchestra.run.smithi063.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 950 kB/s | 245 kB 00:00 2024-03-25T16:22:42.661 INFO:teuthology.orchestra.run.smithi063.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 624 kB/s | 7.4 MB 00:12 2024-03-25T16:22:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:Total 3.0 MB/s | 37 MB 00:12 2024-03-25T16:22:42.708 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:22:42.735 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:22:42.735 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:22:43.115 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:22:43.115 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:22:43.843 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:22:43.901 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:22:43.923 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:22:44.051 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:22:44.085 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:22:44.263 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:22:44.300 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:22:44.384 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:22:44.516 INFO:teuthology.orchestra.run.smithi081.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 928 kB/s | 22 MB 00:24 2024-03-25T16:22:45.079 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:22:45.200 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:22:46.506 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:22:47.597 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:22:47.633 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:22:50.186 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:22:50.186 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:22:50.186 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:22:50.186 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:22:50.186 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:22:50.186 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:22:50.186 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:22:50.187 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:22:50.187 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:22:50.187 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:22:50.187 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:22:50.187 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:22:50.611 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:22:50.611 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:50.611 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:22:50.611 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:50.611 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:22:50.611 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:22:50.611 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:22:50.611 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:22:50.611 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:22:50.611 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:22:50.612 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:22:50.612 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:22:50.612 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:22:50.612 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:22:50.612 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:22:50.612 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:50.612 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:22:50.994 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:22:51.577 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:03:58 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:51.735 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:22:51.736 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T16:22:51.737 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T16:22:51.737 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T16:22:51.737 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:22:51.737 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:22:51.737 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T16:22:51.737 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:22:51.737 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:22:51.737 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:22:51.737 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:22:51.737 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:22:51.737 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:22:51.737 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:22:51.737 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:22:51.738 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:22:51.738 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:22:51.738 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:22:51.738 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:22:51.738 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:51.738 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:22:51.738 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T16:22:51.738 INFO:teuthology.orchestra.run.smithi063.stdout:Install 13 Packages 2024-03-25T16:22:51.738 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:51.739 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 2.0 M 2024-03-25T16:22:51.739 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 26 M 2024-03-25T16:22:51.739 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:22:52.033 INFO:teuthology.orchestra.run.smithi063.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 222 kB/s | 24 kB 00:00 2024-03-25T16:22:52.082 INFO:teuthology.orchestra.run.smithi063.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 113 kB/s | 18 kB 00:00 2024-03-25T16:22:52.124 INFO:teuthology.orchestra.run.smithi063.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-25T16:22:52.141 INFO:teuthology.orchestra.run.smithi063.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-25T16:22:52.167 INFO:teuthology.orchestra.run.smithi063.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-03-25T16:22:52.191 INFO:teuthology.orchestra.run.smithi063.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-25T16:22:52.442 INFO:teuthology.orchestra.run.smithi063.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-03-25T16:22:52.467 INFO:teuthology.orchestra.run.smithi063.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-03-25T16:22:52.642 INFO:teuthology.orchestra.run.smithi063.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 72 kB/s | 52 kB 00:00 2024-03-25T16:22:52.818 INFO:teuthology.orchestra.run.smithi063.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 540 kB/s | 203 kB 00:00 2024-03-25T16:22:52.943 INFO:teuthology.orchestra.run.smithi063.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 719 kB/s | 90 kB 00:00 2024-03-25T16:22:53.009 INFO:teuthology.orchestra.run.smithi063.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 161 kB/s | 59 kB 00:00 2024-03-25T16:22:53.043 INFO:teuthology.orchestra.run.smithi063.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2024-03-25T16:22:53.045 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:53.045 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-25T16:22:53.046 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:22:53.062 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:22:53.062 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:22:53.165 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:22:53.166 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:22:53.702 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:22:53.873 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:22:53.928 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:22:53.985 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:22:54.020 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:22:54.063 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:22:54.161 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:22:54.269 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:22:54.322 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:22:55.091 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:22:55.135 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:22:55.169 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:22:55.218 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:22:55.255 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:22:56.017 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:22:56.017 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:22:56.017 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:22:56.017 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:22:56.017 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:22:56.018 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:22:56.018 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:22:56.018 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:22:56.018 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:22:56.018 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:22:56.018 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:22:56.018 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:22:56.019 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:22:56.375 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:22:56.375 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:56.375 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:22:56.375 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:56.375 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:22:56.375 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:22:56.375 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:22:56.375 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:22:56.375 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:22:56.375 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:22:56.375 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:22:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:22:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:22:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:22:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:22:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:22:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:56.376 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:22:56.643 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:22:57.218 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:04:04 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:22:57.373 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:22:57.374 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T16:22:57.374 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T16:22:57.374 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T16:22:57.374 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:22:57.374 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:22:57.374 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T16:22:57.375 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:22:57.375 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:22:57.375 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:22:57.375 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:22:57.375 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:22:57.375 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:57.375 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:22:57.375 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T16:22:57.375 INFO:teuthology.orchestra.run.smithi063.stdout:Install 6 Packages 2024-03-25T16:22:57.376 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:22:57.376 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 6.8 M 2024-03-25T16:22:57.376 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 33 M 2024-03-25T16:22:57.377 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:22:57.744 INFO:teuthology.orchestra.run.smithi063.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-25T16:22:57.827 INFO:teuthology.orchestra.run.smithi063.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-25T16:22:58.478 INFO:teuthology.orchestra.run.smithi063.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 147 kB/s | 138 kB 00:00 2024-03-25T16:23:01.378 INFO:teuthology.orchestra.run.smithi063.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 16 kB/s | 58 kB 00:03 2024-03-25T16:23:01.762 INFO:teuthology.orchestra.run.smithi063.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 16 kB/s | 51 kB 00:03 2024-03-25T16:23:29.875 INFO:teuthology.orchestra.run.smithi063.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 17 kB/s | 548 kB 00:32 2024-03-25T16:23:29.877 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:29.877 INFO:teuthology.orchestra.run.smithi063.stdout:Total 214 kB/s | 6.8 MB 00:32 2024-03-25T16:23:29.891 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:23:29.904 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:23:29.904 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:23:29.999 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:23:29.999 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:23:30.447 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:23:30.624 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:23:30.677 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:23:31.232 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:23:31.317 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:23:31.385 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:23:31.415 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:23:32.158 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:23:32.158 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:23:32.158 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:23:32.159 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:23:32.159 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:23:32.159 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:23:32.469 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:23:32.469 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:32.469 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:23:32.469 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:32.470 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:23:32.470 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:23:32.470 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:23:32.470 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:23:32.470 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:23:32.470 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:32.470 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:23:32.716 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-fuse 2024-03-25T16:23:33.291 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:04:40 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:23:33.448 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:23:33.448 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T16:23:33.449 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:33.449 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:23:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:23:33.449 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T16:23:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:23:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:33.449 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:23:33.449 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:33.449 INFO:teuthology.orchestra.run.smithi063.stdout:Install 2 Packages 2024-03-25T16:23:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:33.450 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 964 k 2024-03-25T16:23:33.450 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 2.8 M 2024-03-25T16:23:33.450 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:23:33.603 INFO:teuthology.orchestra.run.smithi063.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 515 kB/s | 79 kB 00:00 2024-03-25T16:23:35.036 INFO:teuthology.orchestra.run.smithi063.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 558 kB/s | 885 kB 00:01 2024-03-25T16:23:35.038 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:35.038 INFO:teuthology.orchestra.run.smithi063.stdout:Total 606 kB/s | 964 kB 00:01 2024-03-25T16:23:35.038 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:23:35.064 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:23:35.064 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:23:35.144 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:23:35.145 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:23:35.328 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:23:35.466 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:23:35.531 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:23:36.197 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:23:36.198 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:23:36.496 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:23:36.497 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:36.497 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:23:36.497 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:23:36.497 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:36.497 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:23:36.795 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-volume 2024-03-25T16:23:37.371 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:04:44 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:23:37.527 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:23:37.528 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:37.529 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T16:23:37.529 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:37.529 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:23:37.529 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:23:37.529 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T16:23:37.529 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:23:37.529 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:37.529 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:23:37.529 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:37.529 INFO:teuthology.orchestra.run.smithi063.stdout:Install 2 Packages 2024-03-25T16:23:37.530 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:37.530 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 332 k 2024-03-25T16:23:37.530 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 1.5 M 2024-03-25T16:23:37.530 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:23:37.640 INFO:teuthology.orchestra.run.smithi063.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 620 kB/s | 69 kB 00:00 2024-03-25T16:23:38.332 INFO:teuthology.orchestra.run.smithi063.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 328 kB/s | 264 kB 00:00 2024-03-25T16:23:38.333 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:38.334 INFO:teuthology.orchestra.run.smithi063.stdout:Total 413 kB/s | 332 kB 00:00 2024-03-25T16:23:38.334 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:23:38.344 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:23:38.344 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:23:38.387 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:23:38.387 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:23:38.536 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:23:38.722 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:23:38.759 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:23:38.774 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:23:38.774 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:38.774 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:23:38.774 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:39.565 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:23:39.901 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:23:39.902 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:39.902 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:23:39.902 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:39.902 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:23:39.902 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:39.902 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:23:40.238 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados-devel 2024-03-25T16:23:40.823 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:04:47 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:23:40.978 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:23:40.979 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:40.979 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T16:23:40.979 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:40.979 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:23:40.980 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:23:40.980 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:40.980 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:23:40.980 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:40.980 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T16:23:40.980 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:40.980 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 127 k 2024-03-25T16:23:40.980 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 456 k 2024-03-25T16:23:40.980 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:23:41.579 INFO:teuthology.orchestra.run.smithi063.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 212 kB/s | 127 kB 00:00 2024-03-25T16:23:41.579 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:41.580 INFO:teuthology.orchestra.run.smithi063.stdout:Total 211 kB/s | 127 kB 00:00 2024-03-25T16:23:41.580 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:23:41.585 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:23:41.586 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:23:41.644 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:23:41.645 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:23:41.799 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:23:41.914 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:42.376 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:42.668 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:42.668 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:42.669 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:23:42.669 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:42.669 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:42.669 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:23:42.888 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs2 2024-03-25T16:23:43.467 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:04:50 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:23:43.566 INFO:teuthology.orchestra.run.smithi063.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:43.623 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:23:43.624 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T16:23:43.624 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:23:43.695 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs-devel 2024-03-25T16:23:44.274 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:04:51 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:23:44.432 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:23:44.433 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:44.433 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T16:23:44.433 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:44.433 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:23:44.433 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:23:44.433 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:44.433 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:23:44.433 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:44.433 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T16:23:44.433 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:44.434 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 31 k 2024-03-25T16:23:44.434 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 140 k 2024-03-25T16:23:44.434 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:23:44.890 INFO:teuthology.orchestra.run.smithi063.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 68 kB/s | 31 kB 00:00 2024-03-25T16:23:44.891 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:44.891 INFO:teuthology.orchestra.run.smithi063.stdout:Total 68 kB/s | 31 kB 00:00 2024-03-25T16:23:44.891 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:23:44.895 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:23:44.895 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:23:44.921 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:23:44.921 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:23:45.053 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:23:45.150 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:45.509 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:45.769 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:45.769 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:45.769 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:23:45.769 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:45.769 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:45.769 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:23:45.932 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados2 2024-03-25T16:23:46.507 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:04:53 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:23:46.608 INFO:teuthology.orchestra.run.smithi063.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:46.662 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:23:46.663 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T16:23:46.663 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:23:46.720 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librbd1 2024-03-25T16:23:47.301 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:04:54 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:23:47.402 INFO:teuthology.orchestra.run.smithi063.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:47.457 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:23:47.458 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T16:23:47.458 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:23:47.509 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rados 2024-03-25T16:23:48.084 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:04:55 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:23:48.184 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:48.239 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:23:48.239 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T16:23:48.239 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:23:48.292 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rgw 2024-03-25T16:23:48.870 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:04:55 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:23:48.969 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:49.023 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:23:49.025 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T16:23:49.025 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:23:49.076 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-cephfs 2024-03-25T16:23:49.656 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:04:56 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:23:49.756 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:49.811 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:23:49.812 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T16:23:49.812 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:23:49.864 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rbd 2024-03-25T16:23:50.439 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:04:57 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:23:50.540 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:50.595 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:23:50.596 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T16:23:50.596 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:23:50.651 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-fuse 2024-03-25T16:23:51.228 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:04:58 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:23:51.384 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:23:51.385 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:51.385 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-03-25T16:23:51.385 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:51.385 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:23:51.385 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:23:51.385 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:51.385 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:23:51.385 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:51.385 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T16:23:51.385 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:51.385 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 85 k 2024-03-25T16:23:51.385 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 231 k 2024-03-25T16:23:51.386 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:23:51.918 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 160 kB/s | 85 kB 00:00 2024-03-25T16:23:51.918 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:51.918 INFO:teuthology.orchestra.run.smithi063.stdout:Total 160 kB/s | 85 kB 00:00 2024-03-25T16:23:51.918 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:23:51.926 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:23:51.926 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:23:51.984 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:23:51.985 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:23:52.129 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:23:52.235 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:52.601 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:52.884 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:52.884 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:52.884 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:23:52.884 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:52.884 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:52.884 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:23:53.128 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-mirror 2024-03-25T16:23:53.710 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:05:00 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:23:53.863 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:23:53.864 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:53.864 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T16:23:53.864 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:53.864 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:23:53.864 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:23:53.864 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:53.864 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:23:53.864 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:23:53.864 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T16:23:53.864 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:23:53.865 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 3.1 M 2024-03-25T16:23:53.865 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 13 M 2024-03-25T16:23:53.865 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:23:59.078 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 606 kB/s | 3.1 MB 00:05 2024-03-25T16:23:59.079 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:59.079 INFO:teuthology.orchestra.run.smithi063.stdout:Total 606 kB/s | 3.1 MB 00:05 2024-03-25T16:23:59.080 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:23:59.091 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:23:59.092 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:23:59.172 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:23:59.173 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:23:59.626 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:23:59.695 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:59.719 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:59.720 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:59.720 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:23:59.720 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:23:59.720 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:23:59.720 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:24:00.806 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:00.806 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:24:00.807 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:24:00.807 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:00.807 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:24:00.807 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:24:01.051 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-nbd 2024-03-25T16:24:01.640 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:05:08 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:24:01.797 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:24:01.798 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:24:01.798 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-03-25T16:24:01.798 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:24:01.798 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:24:01.798 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:24:01.798 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:24:01.798 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:24:01.799 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:24:01.799 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T16:24:01.799 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:24:01.799 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 171 k 2024-03-25T16:24:01.799 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 494 k 2024-03-25T16:24:01.799 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:24:02.731 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 184 kB/s | 171 kB 00:00 2024-03-25T16:24:02.732 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:02.732 INFO:teuthology.orchestra.run.smithi063.stdout:Total 184 kB/s | 171 kB 00:00 2024-03-25T16:24:02.733 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:24:02.741 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:24:02.742 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:24:02.800 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:24:02.800 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:24:02.947 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:24:03.041 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:03.487 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:03.790 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:03.790 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:24:03.790 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:24:03.790 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:03.790 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:24:03.790 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:24:04.033 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:24:04.608 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:05:11 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:24:04.766 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:24:04.767 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:24:04.767 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T16:24:04.767 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:24:04.767 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:24:04.767 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:24:04.767 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T16:24:04.767 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:24:04.767 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:24:04.767 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:24:04.767 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:24:04.767 INFO:teuthology.orchestra.run.smithi063.stdout:Install 2 Packages 2024-03-25T16:24:04.767 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:24:04.768 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 91 M 2024-03-25T16:24:04.768 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 498 M 2024-03-25T16:24:04.768 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:24:06.807 INFO:teuthology.orchestra.run.smithi081.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 652 kB/s | 68 MB 01:47 2024-03-25T16:24:06.808 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:06.809 INFO:teuthology.orchestra.run.smithi081.stdout:Total 866 kB/s | 91 MB 01:47 2024-03-25T16:24:06.809 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:24:06.813 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:24:06.814 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:24:07.389 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:24:07.390 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:24:10.494 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:24:17.587 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:24:17.666 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:24:21.046 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:24:21.046 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:24:21.301 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:24:21.301 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:24:21.301 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:24:21.301 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:21.301 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:21.302 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:24:21.302 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:24:21.513 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:24:22.076 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:05:29 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:24:22.224 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:24:22.225 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:24:22.225 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T16:24:22.225 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:24:22.225 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:24:22.225 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:24:22.225 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:24:22.225 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:24:22.225 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:24:22.226 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T16:24:22.226 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:24:22.226 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 416 M 2024-03-25T16:24:22.226 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 1.7 G 2024-03-25T16:24:22.226 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:25:09.254 INFO:teuthology.orchestra.run.smithi063.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 356 kB/s | 22 MB 01:04 2024-03-25T16:25:41.484 INFO:teuthology.orchestra.run.smithi063.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 721 kB/s | 68 MB 01:36 2024-03-25T16:25:41.485 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:41.485 INFO:teuthology.orchestra.run.smithi063.stdout:Total 959 kB/s | 91 MB 01:36 2024-03-25T16:25:41.485 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:25:41.489 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:25:41.490 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:25:42.103 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:25:42.103 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:25:45.493 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:25:53.033 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:25:53.103 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:25:56.299 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:25:56.299 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:25:56.568 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:25:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:25:56.569 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:25:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:25:56.569 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:25:56.810 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:25:57.383 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:07:04 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:25:57.538 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:25:57.541 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:25:57.542 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T16:25:57.542 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:25:57.542 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:25:57.542 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:25:57.542 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:25:57.542 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:25:57.542 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:25:57.542 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T16:25:57.542 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:25:57.542 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 416 M 2024-03-25T16:25:57.542 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 1.7 G 2024-03-25T16:25:57.543 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:46:34.509 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 320 kB/s | 416 MB 22:12 2024-03-25T16:46:34.526 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:46:34.526 INFO:teuthology.orchestra.run.smithi081.stdout:Total 320 kB/s | 416 MB 22:12 2024-03-25T16:46:34.526 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:46:34.527 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:46:34.527 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:46:36.978 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:46:36.978 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:47:19.037 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:47:19.136 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:47:24.202 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:47:24.462 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:47:24.462 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:47:24.462 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:47:24.462 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:47:24.462 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:47:24.462 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:47:24.700 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:47:25.273 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:28:32 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:47:25.432 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:47:25.432 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T16:47:25.433 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T16:47:25.433 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T16:47:25.433 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:47:25.433 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:47:25.433 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:47:25.433 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:47:25.433 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T16:47:25.433 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T16:47:25.433 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:47:25.433 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 1.4 M 2024-03-25T16:47:25.433 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 6.4 M 2024-03-25T16:47:25.433 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:47:32.166 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 217 kB/s | 1.4 MB 00:06 2024-03-25T16:47:32.167 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:47:32.167 INFO:teuthology.orchestra.run.smithi081.stdout:Total 217 kB/s | 1.4 MB 00:06 2024-03-25T16:47:32.167 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:47:32.171 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:47:32.172 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:47:32.216 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:47:32.217 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:47:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:47:32.564 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:47:32.980 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:47:33.253 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:47:33.253 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:47:33.253 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:47:33.254 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:47:33.254 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:47:33.254 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:47:33.449 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:47:34.008 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:28:41 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:47:34.154 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:47:34.155 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:47:34.155 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T16:47:34.155 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:47:34.156 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:47:34.156 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:47:34.156 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:47:34.156 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:47:34.156 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:47:34.156 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T16:47:34.156 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:47:34.156 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 176 M 2024-03-25T16:47:34.156 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 753 M 2024-03-25T16:47:34.156 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:48:11.482 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 319 kB/s | 416 MB 22:13 2024-03-25T16:48:11.482 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:48:11.482 INFO:teuthology.orchestra.run.smithi063.stdout:Total 319 kB/s | 416 MB 22:13 2024-03-25T16:48:11.482 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:48:11.485 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:48:11.486 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:48:14.072 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:48:14.072 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:48:58.691 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:48:58.788 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:49:05.686 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:49:05.942 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:49:05.942 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:49:05.943 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:49:05.943 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:49:05.943 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:49:05.943 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:49:06.286 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:49:06.865 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:30:13 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:49:07.019 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:49:07.020 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T16:49:07.020 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T16:49:07.021 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T16:49:07.021 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:49:07.021 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:49:07.021 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:49:07.021 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:49:07.021 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T16:49:07.021 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T16:49:07.021 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:49:07.021 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 1.4 M 2024-03-25T16:49:07.021 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 6.4 M 2024-03-25T16:49:07.021 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:49:14.504 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 195 kB/s | 1.4 MB 00:07 2024-03-25T16:49:14.505 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:49:14.505 INFO:teuthology.orchestra.run.smithi063.stdout:Total 195 kB/s | 1.4 MB 00:07 2024-03-25T16:49:14.505 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:49:14.509 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:49:14.509 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:49:14.556 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:49:14.556 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T16:49:14.823 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T16:49:14.922 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:49:15.330 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:49:15.602 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:49:15.602 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:49:15.602 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T16:49:15.603 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:49:15.603 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:49:15.603 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T16:49:15.760 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:49:16.334 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:30:23 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:49:16.488 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T16:49:16.489 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:49:16.489 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T16:49:16.489 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:49:16.490 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T16:49:16.490 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:49:16.490 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:49:16.490 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T16:49:16.490 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T16:49:16.490 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T16:49:16.490 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T16:49:16.490 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 176 M 2024-03-25T16:49:16.490 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 753 M 2024-03-25T16:49:16.490 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T16:58:39.655 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 270 kB/s | 176 MB 11:05 2024-03-25T16:58:39.655 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:58:39.655 INFO:teuthology.orchestra.run.smithi081.stdout:Total 270 kB/s | 176 MB 11:05 2024-03-25T16:58:39.656 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T16:58:39.658 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T16:58:39.659 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T16:58:40.716 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T16:58:40.716 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T16:58:58.909 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T16:58:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:59:03.712 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:59:03.978 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:59:03.978 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:59:03.978 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T16:59:03.979 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:59:03.979 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:59:03.979 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T16:59:04.201 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:59:04.763 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:40:11 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T16:59:04.911 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T16:59:04.911 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:59:04.911 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T16:59:04.912 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:59:04.912 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T16:59:04.912 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:59:04.912 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:59:04.912 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T16:59:04.912 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T16:59:04.912 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T16:59:04.912 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T16:59:04.912 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 672 M 2024-03-25T16:59:04.913 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 3.1 G 2024-03-25T16:59:04.913 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T16:59:47.832 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 285 kB/s | 176 MB 10:31 2024-03-25T16:59:47.832 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:59:47.833 INFO:teuthology.orchestra.run.smithi063.stdout:Total 285 kB/s | 176 MB 10:31 2024-03-25T16:59:47.833 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T16:59:47.836 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T16:59:47.836 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T16:59:48.945 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T16:59:48.946 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T17:00:08.027 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T17:00:08.148 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:00:12.725 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:00:12.983 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:00:12.983 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:00:12.983 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T17:00:12.984 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:00:12.984 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:00:12.984 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T17:00:13.196 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:00:13.770 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:41:20 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:00:13.923 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T17:00:13.924 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T17:00:13.924 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T17:00:13.924 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T17:00:13.924 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T17:00:13.924 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:00:13.925 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:00:13.925 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T17:00:13.925 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T17:00:13.925 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T17:00:13.925 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:00:13.925 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 672 M 2024-03-25T17:00:13.925 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 3.1 G 2024-03-25T17:00:13.926 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T17:30:38.649 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 363 kB/s | 672 MB 31:33 2024-03-25T17:30:38.674 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:30:38.674 INFO:teuthology.orchestra.run.smithi081.stdout:Total 363 kB/s | 672 MB 31:33 2024-03-25T17:30:38.674 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:30:38.674 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:30:38.675 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:30:42.602 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:30:42.602 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:31:53.222 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:31:53.424 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:32:02.715 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:32:02.969 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:32:02.969 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:32:02.969 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:32:02.969 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:32:02.969 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:32:02.969 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:32:03.249 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:32:03.811 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:13:10 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:32:03.906 INFO:teuthology.orchestra.run.smithi081.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:32:03.959 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:32:03.960 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T17:32:03.960 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:32:04.031 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:32:04.593 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:13:11 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:32:04.739 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:32:04.740 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:32:04.740 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T17:32:04.740 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:32:04.740 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T17:32:04.740 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:32:04.740 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:32:04.740 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T17:32:04.741 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:32:04.741 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T17:32:04.741 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:32:04.741 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 20 M 2024-03-25T17:32:04.741 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 103 M 2024-03-25T17:32:04.741 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T17:32:37.134 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 623 kB/s | 20 MB 00:32 2024-03-25T17:32:37.134 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:32:37.135 INFO:teuthology.orchestra.run.smithi081.stdout:Total 623 kB/s | 20 MB 00:32 2024-03-25T17:32:37.135 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:32:37.139 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:32:37.139 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:32:37.289 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:32:37.289 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:32:39.511 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:32:39.616 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:32:40.632 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:32:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:32:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:32:40.890 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:32:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:32:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:32:40.891 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:32:41.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:32:41.671 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:13:48 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:32:41.828 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:32:41.829 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:32:41.829 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T17:32:41.829 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:32:41.829 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T17:32:41.829 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:32:41.829 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:32:41.829 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T17:32:41.829 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:32:41.829 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T17:32:41.830 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:32:41.830 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 48 M 2024-03-25T17:32:41.830 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 173 M 2024-03-25T17:32:41.830 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T17:34:03.979 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 595 kB/s | 48 MB 01:22 2024-03-25T17:34:03.979 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:34:03.980 INFO:teuthology.orchestra.run.smithi081.stdout:Total 595 kB/s | 48 MB 01:22 2024-03-25T17:34:03.980 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:34:03.983 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:34:03.984 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:34:04.295 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:34:04.295 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:34:09.029 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:34:09.143 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:34:10.520 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:34:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:34:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:34:10.777 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:34:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:34:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:34:10.778 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:34:10.964 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:34:11.526 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:15:18 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:34:11.674 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:34:11.675 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:34:11.675 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T17:34:11.675 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:34:11.675 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T17:34:11.675 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:34:11.675 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:34:11.675 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T17:34:11.675 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:34:11.676 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T17:34:11.676 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:34:11.676 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 99 M 2024-03-25T17:34:11.676 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 449 M 2024-03-25T17:34:11.676 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T17:36:41.965 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 671 kB/s | 99 MB 02:30 2024-03-25T17:36:41.966 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:36:41.966 INFO:teuthology.orchestra.run.smithi081.stdout:Total 671 kB/s | 99 MB 02:30 2024-03-25T17:36:41.966 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:36:41.969 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:36:41.969 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:36:42.577 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:36:42.577 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:36:52.847 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:36:52.946 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:36:55.886 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:36:56.150 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:36:56.150 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:36:56.150 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:36:56.150 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:36:56.150 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:36:56.150 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:36:56.328 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:36:56.888 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:18:03 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:36:57.037 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:36:57.037 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:36:57.038 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T17:36:57.038 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:36:57.038 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T17:36:57.038 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:36:57.038 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:36:57.038 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T17:36:57.038 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:36:57.038 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T17:36:57.038 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:36:57.038 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 287 M 2024-03-25T17:36:57.038 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 1.2 G 2024-03-25T17:36:57.039 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T17:42:25.905 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 272 kB/s | 672 MB 42:11 2024-03-25T17:42:25.906 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:42:25.906 INFO:teuthology.orchestra.run.smithi063.stdout:Total 272 kB/s | 672 MB 42:11 2024-03-25T17:42:25.906 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T17:42:25.910 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T17:42:25.910 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T17:42:30.065 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T17:42:30.065 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T17:43:45.196 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T17:43:45.302 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:43:53.890 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:43:54.145 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:43:54.145 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:43:54.145 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T17:43:54.145 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:43:54.145 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:43:54.145 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T17:43:54.465 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:43:55.041 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:25:02 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:43:55.140 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:43:55.195 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T17:43:55.196 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T17:43:55.197 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T17:43:55.266 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:43:55.843 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:25:02 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:43:55.997 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T17:43:55.998 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T17:43:55.998 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T17:43:55.999 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T17:43:55.999 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T17:43:55.999 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:43:55.999 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:43:55.999 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T17:43:55.999 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T17:43:55.999 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T17:43:55.999 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:43:55.999 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 20 M 2024-03-25T17:43:55.999 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 103 M 2024-03-25T17:43:55.999 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T17:44:53.346 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 352 kB/s | 20 MB 00:57 2024-03-25T17:44:53.347 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:44:53.348 INFO:teuthology.orchestra.run.smithi063.stdout:Total 352 kB/s | 20 MB 00:57 2024-03-25T17:44:53.348 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T17:44:53.351 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T17:44:53.351 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T17:44:53.506 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T17:44:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T17:44:55.864 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T17:44:55.973 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:56.939 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:57.194 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:57.194 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:44:57.194 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T17:44:57.194 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:44:57.194 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:44:57.194 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T17:44:57.410 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:44:57.994 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:26:04 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:44:58.155 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T17:44:58.156 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T17:44:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T17:44:58.156 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T17:44:58.156 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T17:44:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:44:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:44:58.156 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T17:44:58.157 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T17:44:58.157 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T17:44:58.157 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:44:58.157 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 48 M 2024-03-25T17:44:58.157 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 173 M 2024-03-25T17:44:58.157 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T17:46:18.816 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 606 kB/s | 48 MB 01:20 2024-03-25T17:46:18.817 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:46:18.817 INFO:teuthology.orchestra.run.smithi063.stdout:Total 606 kB/s | 48 MB 01:20 2024-03-25T17:46:18.817 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T17:46:18.820 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T17:46:18.820 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T17:46:19.147 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T17:46:19.147 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T17:46:24.183 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T17:46:24.282 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:46:25.556 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:46:25.816 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:46:25.816 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:46:25.816 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T17:46:25.816 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:46:25.816 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:46:25.817 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T17:46:25.980 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:46:26.556 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:27:33 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:46:26.710 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T17:46:26.711 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T17:46:26.711 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T17:46:26.711 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T17:46:26.711 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T17:46:26.711 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:46:26.711 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:46:26.711 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T17:46:26.711 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T17:46:26.711 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T17:46:26.711 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:46:26.712 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 99 M 2024-03-25T17:46:26.712 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 449 M 2024-03-25T17:46:26.712 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T17:48:24.818 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 427 kB/s | 287 MB 11:27 2024-03-25T17:48:24.818 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:48:24.818 INFO:teuthology.orchestra.run.smithi081.stdout:Total 427 kB/s | 287 MB 11:27 2024-03-25T17:48:24.819 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:48:24.821 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:48:24.821 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:48:26.522 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:48:26.522 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:48:55.815 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:48:56.357 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:57.810 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:58.093 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:58.093 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:48:58.093 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:48:58.093 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:58.094 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:48:58.094 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:48:58.288 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:48:58.849 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:30:05 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:48:58.996 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:48:58.997 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:48:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T17:48:58.997 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:48:58.997 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T17:48:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:48:58.998 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:48:58.998 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T17:48:58.998 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:48:58.998 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T17:48:58.998 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:48:58.998 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 5.9 M 2024-03-25T17:48:58.998 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 28 M 2024-03-25T17:48:58.998 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T17:49:21.249 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 274 kB/s | 5.9 MB 00:22 2024-03-25T17:49:21.249 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:21.250 INFO:teuthology.orchestra.run.smithi081.stdout:Total 273 kB/s | 5.9 MB 00:22 2024-03-25T17:49:21.250 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:49:21.254 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:49:21.254 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:49:21.324 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:49:21.325 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:49:22.056 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:49:22.155 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:49:22.759 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:49:23.019 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:49:23.019 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:49:23.019 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:49:23.019 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:23.019 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:49:23.019 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:49:23.215 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:49:23.772 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:30:30 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:49:23.925 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:49:23.925 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:49:23.925 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T17:49:23.925 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:49:23.925 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T17:49:23.925 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:49:23.925 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:49:23.926 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T17:49:23.926 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:49:23.926 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T17:49:23.926 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:49:23.926 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 109 k 2024-03-25T17:49:23.926 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 440 k 2024-03-25T17:49:23.926 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T17:49:24.763 INFO:teuthology.orchestra.run.smithi081.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 130 kB/s | 109 kB 00:00 2024-03-25T17:49:24.763 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:24.764 INFO:teuthology.orchestra.run.smithi081.stdout:Total 130 kB/s | 109 kB 00:00 2024-03-25T17:49:24.764 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:49:24.767 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:49:24.767 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:49:24.802 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:49:24.803 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:49:24.922 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:49:25.056 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:49:25.427 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:49:25.698 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:49:25.698 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:49:25.698 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:49:25.698 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:25.698 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:49:25.698 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:49:25.891 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:49:26.452 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:30:33 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:49:26.599 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:49:26.600 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:49:26.600 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T17:49:26.600 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:49:26.601 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T17:49:26.601 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:49:26.601 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:49:26.601 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T17:49:26.601 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:49:26.601 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T17:49:26.601 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:49:26.601 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 5.4 M 2024-03-25T17:49:26.601 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 25 M 2024-03-25T17:49:26.601 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T17:49:46.013 INFO:teuthology.orchestra.run.smithi081.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 282 kB/s | 5.4 MB 00:19 2024-03-25T17:49:46.014 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:46.014 INFO:teuthology.orchestra.run.smithi081.stdout:Total 282 kB/s | 5.4 MB 00:19 2024-03-25T17:49:46.015 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:49:46.018 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:49:46.019 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:49:46.086 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:49:46.086 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:49:46.753 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:49:46.853 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:49:47.419 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:49:47.682 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:49:47.682 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:49:47.682 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:49:47.682 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:47.682 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:49:47.682 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:49:47.876 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librados2-debuginfo 2024-03-25T17:49:48.439 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:30:55 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:49:48.587 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:49:48.588 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:49:48.588 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T17:49:48.588 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:49:48.588 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T17:49:48.588 INFO:teuthology.orchestra.run.smithi081.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:49:48.588 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:49:48.588 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T17:49:48.589 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:49:48.589 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T17:49:48.589 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:49:48.589 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 44 M 2024-03-25T17:49:48.589 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 253 M 2024-03-25T17:49:48.589 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T17:51:17.463 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 347 kB/s | 99 MB 04:50 2024-03-25T17:51:17.463 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:17.463 INFO:teuthology.orchestra.run.smithi063.stdout:Total 347 kB/s | 99 MB 04:50 2024-03-25T17:51:17.464 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T17:51:17.467 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T17:51:17.467 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T17:51:18.104 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T17:51:18.104 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T17:51:29.137 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T17:51:29.246 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:32.153 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:32.419 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:32.419 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:51:32.419 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T17:51:32.419 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:32.419 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:51:32.419 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T17:51:32.589 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:51:33.165 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:32:40 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:51:33.319 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T17:51:33.320 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T17:51:33.320 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T17:51:33.320 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T17:51:33.320 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T17:51:33.320 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:51:33.320 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:51:33.320 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T17:51:33.320 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T17:51:33.320 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T17:51:33.320 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T17:51:33.321 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 287 M 2024-03-25T17:51:33.321 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 1.2 G 2024-03-25T17:51:33.321 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T17:52:14.709 INFO:teuthology.orchestra.run.smithi081.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 310 kB/s | 44 MB 02:26 2024-03-25T17:52:14.710 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:14.710 INFO:teuthology.orchestra.run.smithi081.stdout:Total 310 kB/s | 44 MB 02:26 2024-03-25T17:52:14.710 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:52:14.714 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:52:14.714 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:52:15.011 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:52:15.012 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:52:20.035 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:52:20.133 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:52:21.967 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:52:22.225 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:52:22.225 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:52:22.225 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:52:22.225 INFO:teuthology.orchestra.run.smithi081.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:22.225 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:52:22.226 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:52:22.399 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:52:22.964 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:33:29 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:52:23.114 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:52:23.114 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:52:23.115 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T17:52:23.115 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:52:23.115 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T17:52:23.115 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:52:23.115 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:52:23.115 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T17:52:23.115 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:52:23.115 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T17:52:23.115 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:52:23.115 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 52 M 2024-03-25T17:52:23.115 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 308 M 2024-03-25T17:52:23.115 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T17:54:57.556 INFO:teuthology.orchestra.run.smithi081.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 344 kB/s | 52 MB 02:34 2024-03-25T17:54:57.556 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:57.556 INFO:teuthology.orchestra.run.smithi081.stdout:Total 344 kB/s | 52 MB 02:34 2024-03-25T17:54:57.556 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:54:57.559 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:54:57.559 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:54:57.901 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:54:57.901 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:55:03.871 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:55:04.112 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:55:06.348 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:55:06.614 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:55:06.614 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:06.614 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:55:06.614 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:06.615 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:06.615 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:55:06.808 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:55:07.362 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:36:14 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:55:07.510 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:55:07.511 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:07.511 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T17:55:07.511 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:07.512 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T17:55:07.512 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:55:07.512 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:07.512 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T17:55:07.512 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:07.512 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T17:55:07.512 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:07.512 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 615 k 2024-03-25T17:55:07.512 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 2.1 M 2024-03-25T17:55:07.512 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T17:55:11.024 INFO:teuthology.orchestra.run.smithi081.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 175 kB/s | 615 kB 00:03 2024-03-25T17:55:11.024 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:11.025 INFO:teuthology.orchestra.run.smithi081.stdout:Total 175 kB/s | 615 kB 00:03 2024-03-25T17:55:11.025 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:55:11.029 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:55:11.029 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:55:11.071 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:55:11.071 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:55:11.234 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:55:11.347 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:55:11.782 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:55:12.044 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:55:12.045 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:12.045 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:55:12.045 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:12.045 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:12.045 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:55:12.236 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:55:12.794 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:36:19 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:55:12.942 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:55:12.943 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:12.943 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T17:55:12.943 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:12.943 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T17:55:12.943 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:55:12.943 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:12.943 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T17:55:12.944 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:12.944 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T17:55:12.944 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:12.944 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 771 k 2024-03-25T17:55:12.944 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 2.7 M 2024-03-25T17:55:12.944 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T17:55:15.077 INFO:teuthology.orchestra.run.smithi081.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 361 kB/s | 771 kB 00:02 2024-03-25T17:55:15.077 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:15.078 INFO:teuthology.orchestra.run.smithi081.stdout:Total 361 kB/s | 771 kB 00:02 2024-03-25T17:55:15.078 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:55:15.082 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:55:15.082 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:55:15.124 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:55:15.125 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:55:15.307 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:55:15.400 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:55:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:55:16.077 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:55:16.077 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:16.077 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:55:16.078 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:16.078 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:16.078 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:55:16.299 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:55:16.858 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:36:23 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:55:17.007 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:55:17.008 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:17.008 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T17:55:17.008 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:17.008 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T17:55:17.008 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:55:17.008 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:17.008 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T17:55:17.008 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T17:55:17.009 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 1.1 M 2024-03-25T17:55:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 4.3 M 2024-03-25T17:55:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T17:55:21.572 INFO:teuthology.orchestra.run.smithi081.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 252 kB/s | 1.1 MB 00:04 2024-03-25T17:55:21.572 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:21.572 INFO:teuthology.orchestra.run.smithi081.stdout:Total 252 kB/s | 1.1 MB 00:04 2024-03-25T17:55:21.573 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:55:21.577 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:55:21.577 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:55:21.620 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:55:21.620 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:55:21.837 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:55:21.946 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:55:22.388 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:55:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:55:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:22.644 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:55:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:22.645 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:55:22.855 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:55:23.416 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:36:30 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:55:23.565 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:55:23.566 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T17:55:23.566 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:23.566 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T17:55:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:55:23.567 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:23.567 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T17:55:23.567 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:23.567 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T17:55:23.567 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:23.567 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 337 k 2024-03-25T17:55:23.567 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 1.0 M 2024-03-25T17:55:23.567 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T17:55:25.450 INFO:teuthology.orchestra.run.smithi081.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 179 kB/s | 337 kB 00:01 2024-03-25T17:55:25.450 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:25.451 INFO:teuthology.orchestra.run.smithi081.stdout:Total 179 kB/s | 337 kB 00:01 2024-03-25T17:55:25.451 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:55:25.455 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:55:25.455 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:55:25.494 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:55:25.494 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:55:25.631 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:55:25.765 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:55:26.154 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:55:26.410 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:55:26.410 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:26.410 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:55:26.410 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:26.410 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:26.410 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:55:26.624 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:55:27.182 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:36:34 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:55:27.330 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:55:27.331 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:27.331 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T17:55:27.331 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:27.331 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T17:55:27.331 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:55:27.332 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:27.332 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T17:55:27.332 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:27.332 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T17:55:27.332 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:27.332 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 509 k 2024-03-25T17:55:27.332 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 2.5 M 2024-03-25T17:55:27.332 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T17:55:29.628 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 222 kB/s | 509 kB 00:02 2024-03-25T17:55:29.628 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:29.629 INFO:teuthology.orchestra.run.smithi081.stdout:Total 222 kB/s | 509 kB 00:02 2024-03-25T17:55:29.629 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:55:29.633 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:55:29.633 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:55:29.671 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:55:29.671 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:55:29.827 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:55:29.927 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:30.317 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:30.575 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:30.575 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:30.576 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:55:30.576 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:30.576 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:30.576 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:55:30.775 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:55:31.332 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:36:38 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:55:31.482 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:55:31.483 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:31.483 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T17:55:31.483 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:31.483 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T17:55:31.483 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:55:31.483 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:31.483 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T17:55:31.483 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:55:31.483 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T17:55:31.483 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:55:31.484 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 52 M 2024-03-25T17:55:31.484 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 298 M 2024-03-25T17:55:31.484 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T17:58:10.123 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 333 kB/s | 52 MB 02:38 2024-03-25T17:58:10.124 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:10.124 INFO:teuthology.orchestra.run.smithi081.stdout:Total 333 kB/s | 52 MB 02:38 2024-03-25T17:58:10.124 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:58:10.127 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:58:10.127 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:58:10.461 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:58:10.461 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:58:16.335 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:58:16.437 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:18.505 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:18.775 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:18.775 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:58:18.776 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:58:18.776 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:18.776 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:58:18.776 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:58:18.987 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T17:58:19.546 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 1:39:26 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T17:58:19.694 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T17:58:19.695 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:58:19.695 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T17:58:19.695 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:58:19.695 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T17:58:19.695 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T17:58:19.695 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:58:19.695 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T17:58:19.696 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T17:58:19.696 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T17:58:19.696 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:58:19.696 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 1.3 M 2024-03-25T17:58:19.696 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 5.5 M 2024-03-25T17:58:19.696 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T17:58:23.108 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 376 kB/s | 1.3 MB 00:03 2024-03-25T17:58:23.108 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:23.109 INFO:teuthology.orchestra.run.smithi081.stdout:Total 376 kB/s | 1.3 MB 00:03 2024-03-25T17:58:23.109 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T17:58:23.113 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T17:58:23.113 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T17:58:23.156 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T17:58:23.157 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T17:58:23.391 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T17:58:23.499 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:58:23.947 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:58:24.206 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:58:24.206 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:58:24.206 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T17:58:24.206 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:24.206 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T17:58:24.206 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T17:58:24.412 DEBUG:teuthology.parallel:result is None 2024-03-25T18:00:52.722 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 525 kB/s | 287 MB 09:19 2024-03-25T18:00:52.723 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:52.723 INFO:teuthology.orchestra.run.smithi063.stdout:Total 525 kB/s | 287 MB 09:19 2024-03-25T18:00:52.723 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T18:00:52.725 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T18:00:52.726 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T18:00:54.515 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T18:00:54.515 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T18:01:25.751 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T18:01:25.863 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:01:27.139 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:01:27.407 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:01:27.408 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:01:27.408 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T18:01:27.408 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:27.408 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:01:27.408 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T18:01:27.615 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:01:28.195 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:42:35 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T18:01:28.350 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T18:01:28.351 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:01:28.351 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T18:01:28.351 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:01:28.351 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T18:01:28.351 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:01:28.351 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:01:28.351 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T18:01:28.351 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:01:28.351 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T18:01:28.352 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:01:28.352 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 5.9 M 2024-03-25T18:01:28.352 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 28 M 2024-03-25T18:01:28.352 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T18:01:48.285 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 305 kB/s | 5.9 MB 00:19 2024-03-25T18:01:48.286 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:48.287 INFO:teuthology.orchestra.run.smithi063.stdout:Total 305 kB/s | 5.9 MB 00:19 2024-03-25T18:01:48.287 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T18:01:48.291 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T18:01:48.291 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T18:01:48.367 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T18:01:48.367 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T18:01:49.141 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T18:01:49.256 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:01:49.818 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:01:50.084 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:01:50.084 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:01:50.084 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T18:01:50.084 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:50.084 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:01:50.085 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T18:01:50.286 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:01:50.861 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:42:57 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T18:01:51.016 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T18:01:51.017 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:01:51.017 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T18:01:51.017 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:01:51.017 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T18:01:51.017 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:01:51.017 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:01:51.017 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T18:01:51.017 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:01:51.017 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T18:01:51.017 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:01:51.018 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 109 k 2024-03-25T18:01:51.018 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 440 k 2024-03-25T18:01:51.018 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T18:01:51.956 INFO:teuthology.orchestra.run.smithi063.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 116 kB/s | 109 kB 00:00 2024-03-25T18:01:51.956 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:51.957 INFO:teuthology.orchestra.run.smithi063.stdout:Total 116 kB/s | 109 kB 00:00 2024-03-25T18:01:51.957 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T18:01:51.960 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T18:01:51.961 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T18:01:51.998 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T18:01:51.999 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T18:01:52.125 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T18:01:52.241 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:01:52.637 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:01:52.891 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:01:52.891 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:01:52.891 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T18:01:52.892 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:52.892 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:01:52.892 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T18:01:53.059 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:01:53.633 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:43:00 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T18:01:53.787 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T18:01:53.788 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:01:53.789 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T18:01:53.789 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:01:53.789 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T18:01:53.789 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:01:53.789 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:01:53.789 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T18:01:53.789 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:01:53.789 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T18:01:53.789 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:01:53.789 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 5.4 M 2024-03-25T18:01:53.790 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 25 M 2024-03-25T18:01:53.790 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T18:02:08.340 INFO:teuthology.orchestra.run.smithi063.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 377 kB/s | 5.4 MB 00:14 2024-03-25T18:02:08.341 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:02:08.342 INFO:teuthology.orchestra.run.smithi063.stdout:Total 377 kB/s | 5.4 MB 00:14 2024-03-25T18:02:08.342 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T18:02:08.346 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T18:02:08.346 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T18:02:08.417 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T18:02:08.417 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T18:02:09.115 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T18:02:09.233 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:02:09.766 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:02:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:02:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:02:10.032 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T18:02:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:10.033 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:02:10.033 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T18:02:10.226 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados2-debuginfo 2024-03-25T18:02:10.799 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:43:17 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T18:02:10.954 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T18:02:10.955 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:02:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T18:02:10.955 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:02:10.955 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T18:02:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:02:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:02:10.955 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T18:02:10.955 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:02:10.955 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T18:02:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:02:10.955 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 44 M 2024-03-25T18:02:10.955 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 253 M 2024-03-25T18:02:10.955 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T18:05:29.143 INFO:teuthology.orchestra.run.smithi063.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 229 kB/s | 44 MB 03:18 2024-03-25T18:05:29.143 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:29.143 INFO:teuthology.orchestra.run.smithi063.stdout:Total 229 kB/s | 44 MB 03:18 2024-03-25T18:05:29.143 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T18:05:29.146 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T18:05:29.146 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T18:05:29.458 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T18:05:29.459 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T18:05:34.808 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T18:05:34.901 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:05:36.711 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:05:36.963 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:05:36.963 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:05:36.963 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T18:05:36.963 INFO:teuthology.orchestra.run.smithi063.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:36.963 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:05:36.963 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T18:05:37.264 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:05:37.836 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:46:44 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T18:05:37.993 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T18:05:37.993 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:05:37.993 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T18:05:37.994 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:05:37.994 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T18:05:37.994 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:05:37.994 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:05:37.994 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T18:05:37.994 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:05:37.994 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T18:05:37.994 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:05:37.995 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 52 M 2024-03-25T18:05:37.995 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 308 M 2024-03-25T18:05:37.995 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T18:07:56.548 INFO:teuthology.orchestra.run.smithi063.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 383 kB/s | 52 MB 02:18 2024-03-25T18:07:56.549 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:56.549 INFO:teuthology.orchestra.run.smithi063.stdout:Total 383 kB/s | 52 MB 02:18 2024-03-25T18:07:56.549 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T18:07:56.552 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T18:07:56.552 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T18:07:56.911 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T18:07:56.911 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T18:08:03.283 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T18:08:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:05.463 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:05.755 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:05.755 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:05.755 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T18:08:05.755 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:05.755 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:05.755 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T18:08:05.956 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:08:06.526 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:49:13 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T18:08:06.681 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T18:08:06.682 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:06.682 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T18:08:06.682 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:06.682 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T18:08:06.682 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:08:06.683 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:06.683 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T18:08:06.683 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:06.683 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T18:08:06.683 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:06.683 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 615 k 2024-03-25T18:08:06.683 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 2.1 M 2024-03-25T18:08:06.683 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T18:08:10.333 INFO:teuthology.orchestra.run.smithi063.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 168 kB/s | 615 kB 00:03 2024-03-25T18:08:10.334 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:10.334 INFO:teuthology.orchestra.run.smithi063.stdout:Total 168 kB/s | 615 kB 00:03 2024-03-25T18:08:10.334 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T18:08:10.339 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T18:08:10.339 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T18:08:10.383 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T18:08:10.384 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T18:08:10.556 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T18:08:10.652 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:08:11.032 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:08:11.277 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:08:11.277 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:11.277 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T18:08:11.278 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:11.278 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:11.278 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T18:08:11.466 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:08:12.039 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:49:19 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T18:08:12.194 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T18:08:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:12.195 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T18:08:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T18:08:12.195 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:08:12.195 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T18:08:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T18:08:12.196 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 771 k 2024-03-25T18:08:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 2.7 M 2024-03-25T18:08:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T18:08:15.128 INFO:teuthology.orchestra.run.smithi063.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 263 kB/s | 771 kB 00:02 2024-03-25T18:08:15.128 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:15.129 INFO:teuthology.orchestra.run.smithi063.stdout:Total 263 kB/s | 771 kB 00:02 2024-03-25T18:08:15.129 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T18:08:15.133 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T18:08:15.133 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T18:08:15.175 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T18:08:15.176 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T18:08:15.364 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T18:08:15.471 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:08:15.836 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:08:16.093 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:08:16.093 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:16.094 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T18:08:16.094 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:16.094 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:16.094 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T18:08:16.292 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:08:16.863 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:49:23 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T18:08:17.018 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T18:08:17.019 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:17.019 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T18:08:17.019 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:17.019 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T18:08:17.019 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:08:17.019 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:17.019 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T18:08:17.019 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:17.019 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T18:08:17.019 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:17.019 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 1.1 M 2024-03-25T18:08:17.019 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 4.3 M 2024-03-25T18:08:17.019 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T18:08:21.173 INFO:teuthology.orchestra.run.smithi063.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 277 kB/s | 1.1 MB 00:04 2024-03-25T18:08:21.173 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:21.173 INFO:teuthology.orchestra.run.smithi063.stdout:Total 277 kB/s | 1.1 MB 00:04 2024-03-25T18:08:21.174 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T18:08:21.178 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T18:08:21.178 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T18:08:21.223 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T18:08:21.224 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T18:08:21.457 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T18:08:21.560 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:08:21.939 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:08:22.193 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:08:22.193 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:22.193 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T18:08:22.193 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:22.193 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:22.193 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T18:08:22.387 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:08:22.964 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:49:29 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T18:08:23.121 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T18:08:23.122 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:23.122 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T18:08:23.122 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:23.123 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T18:08:23.123 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:08:23.123 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:23.123 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T18:08:23.123 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:23.123 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T18:08:23.123 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:23.123 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 337 k 2024-03-25T18:08:23.123 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 1.0 M 2024-03-25T18:08:23.123 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T18:08:25.551 INFO:teuthology.orchestra.run.smithi063.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 139 kB/s | 337 kB 00:02 2024-03-25T18:08:25.551 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:25.552 INFO:teuthology.orchestra.run.smithi063.stdout:Total 139 kB/s | 337 kB 00:02 2024-03-25T18:08:25.552 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T18:08:25.556 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T18:08:25.556 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T18:08:25.597 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T18:08:25.597 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T18:08:25.743 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T18:08:25.861 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:08:26.207 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:08:26.468 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:08:26.468 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:26.468 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T18:08:26.469 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:26.469 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:26.469 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T18:08:26.668 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:08:27.243 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:49:34 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T18:08:27.400 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T18:08:27.400 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:27.400 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T18:08:27.400 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:27.400 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T18:08:27.400 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:08:27.401 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:27.401 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T18:08:27.401 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:27.401 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T18:08:27.401 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:27.401 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 509 k 2024-03-25T18:08:27.401 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 2.5 M 2024-03-25T18:08:27.401 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T18:08:30.646 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 157 kB/s | 509 kB 00:03 2024-03-25T18:08:30.646 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:30.646 INFO:teuthology.orchestra.run.smithi063.stdout:Total 157 kB/s | 509 kB 00:03 2024-03-25T18:08:30.647 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T18:08:30.651 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T18:08:30.651 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T18:08:30.692 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T18:08:30.692 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T18:08:30.865 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T18:08:30.981 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:08:31.329 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:08:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:08:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:31.585 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T18:08:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:31.585 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T18:08:31.778 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:08:32.353 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:49:39 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T18:08:32.511 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T18:08:32.511 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:32.512 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T18:08:32.512 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:32.512 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T18:08:32.512 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:08:32.512 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:32.512 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T18:08:32.512 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:08:32.512 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T18:08:32.512 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:08:32.512 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 52 M 2024-03-25T18:08:32.512 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 298 M 2024-03-25T18:08:32.512 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T18:11:28.066 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 301 kB/s | 52 MB 02:55 2024-03-25T18:11:28.066 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:28.066 INFO:teuthology.orchestra.run.smithi063.stdout:Total 301 kB/s | 52 MB 02:55 2024-03-25T18:11:28.067 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T18:11:28.069 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T18:11:28.069 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T18:11:28.420 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T18:11:28.420 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T18:11:34.680 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T18:11:34.792 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:11:36.752 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:11:37.022 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:11:37.022 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:37.022 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T18:11:37.023 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:37.023 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:37.023 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T18:11:37.205 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:11:37.783 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 1:52:44 ago on Mon 25 Mar 2024 04:18:53 PM UTC. 2024-03-25T18:11:37.938 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T18:11:37.939 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:11:37.939 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T18:11:37.939 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:11:37.939 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T18:11:37.939 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:11:37.939 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:37.939 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T18:11:37.939 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T18:11:37.939 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T18:11:37.939 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:37.940 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 1.3 M 2024-03-25T18:11:37.940 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 5.5 M 2024-03-25T18:11:37.940 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T18:11:40.842 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 442 kB/s | 1.3 MB 00:02 2024-03-25T18:11:40.842 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:40.843 INFO:teuthology.orchestra.run.smithi063.stdout:Total 442 kB/s | 1.3 MB 00:02 2024-03-25T18:11:40.843 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T18:11:40.847 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T18:11:40.847 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T18:11:40.894 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T18:11:40.895 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T18:11:41.166 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T18:11:41.278 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:11:41.682 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:11:41.946 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:11:41.946 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:41.946 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T18:11:41.946 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:41.946 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:41.946 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T18:11:42.129 DEBUG:teuthology.parallel:result is None 2024-03-25T18:11:42.129 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:11:42.278 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:11:42.298 INFO:teuthology.orchestra.run.smithi063.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:11:42.299 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:11:42.299 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:11:42.301 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:11:42.515 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:11:42.540 INFO:teuthology.orchestra.run.smithi081.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:11:42.541 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:11:42.541 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:11:42.542 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:11:42.542 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:11:42.543 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:11:42.570 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:11:42.570 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:11:42.612 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:11:42.612 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:11:42.612 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:11:42.640 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:11:42.706 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:11:42.706 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:11:42.736 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:11:42.805 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:11:42.805 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:11:42.805 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:11:42.831 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:11:42.898 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:11:42.898 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:11:42.926 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:11:42.994 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:11:42.994 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:11:42.994 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:11:43.020 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:11:43.086 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:11:43.087 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:11:43.115 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:11:43.184 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:11:43.273 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:11:43.273 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 777 /var/log/ceph 2024-03-25T18:11:43.275 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 777 /var/log/ceph 2024-03-25T18:11:43.302 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:11:43.303 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:11:43.342 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:11:43.369 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:11:43.369 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:11:43.409 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:11:43.468 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:11:43.468 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:11:43.468 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621256', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi063.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCuqPn9P/7bTYjv4OwFl5It97HHvWJqNzhrbIGO9jXyoSczazWUGovbwr+XcXWyTjBXQq5Rnxziuri9JqQFfy2c=', 'smithi081.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBfBt3+3t0WojqxorpmvGW+mvVvJwbVcKMoUxRrsp1ZH3dMxNuK0Q3eqzhXGLQyLCJEcwaE+pFQXqyAv0TkZGFw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:11:43.468 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:11:43.476 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:11:43.491 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:11:43.530 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:11:43.558 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:11:43.558 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:11:43.609 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:11:43.609 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_1 2024-03-25T18:11:43.663 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:11:43.664 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:11:43.664 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T18:11:43.664 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:11:43.664 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:11:43.664 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-25 16:23:59.961179627 +0000 2024-03-25T18:11:43.664 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-25 16:18:05.912004922 +0000 2024-03-25T18:11:43.664 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-25 16:18:05.912004922 +0000 2024-03-25T18:11:43.664 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-25 16:18:05.912004922 +0000 2024-03-25T18:11:43.664 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:11:43.728 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-25T18:11:43.728 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-25T18:11:43.728 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000193969 s, 2.6 MB/s 2024-03-25T18:11:43.729 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:11:43.785 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_2 2024-03-25T18:11:43.838 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:11:43.838 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:11:43.839 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T18:11:43.839 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:11:43.839 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:11:43.839 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-25 16:23:59.962179617 +0000 2024-03-25T18:11:43.839 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-25 16:18:06.270001203 +0000 2024-03-25T18:11:43.839 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-25 16:18:06.270001203 +0000 2024-03-25T18:11:43.839 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-25 16:18:06.270001203 +0000 2024-03-25T18:11:43.839 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:11:43.903 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-25T18:11:43.903 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-25T18:11:43.903 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000168444 s, 3.0 MB/s 2024-03-25T18:11:43.905 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:11:43.959 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_3 2024-03-25T18:11:44.013 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:11:44.013 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:11:44.013 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T18:11:44.014 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:11:44.014 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:11:44.014 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-25 16:23:59.962179617 +0000 2024-03-25T18:11:44.014 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-25 16:18:06.614997619 +0000 2024-03-25T18:11:44.014 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-25 16:18:06.614997619 +0000 2024-03-25T18:11:44.014 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-25 16:18:06.614997619 +0000 2024-03-25T18:11:44.014 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:11:44.078 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-25T18:11:44.079 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-25T18:11:44.079 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000173999 s, 2.9 MB/s 2024-03-25T18:11:44.080 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:11:44.134 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_4 2024-03-25T18:11:44.188 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:11:44.188 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:11:44.188 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T18:11:44.189 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:11:44.189 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:11:44.189 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-25 16:23:59.963179607 +0000 2024-03-25T18:11:44.189 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-25 16:18:06.950994129 +0000 2024-03-25T18:11:44.189 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-25 16:18:06.950994129 +0000 2024-03-25T18:11:44.189 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-25 16:18:06.950994129 +0000 2024-03-25T18:11:44.189 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:11:44.253 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-25T18:11:44.253 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-25T18:11:44.253 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.00016957 s, 3.0 MB/s 2024-03-25T18:11:44.255 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:11:44.309 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:11:44.309 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:11:44.309 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:11:44.326 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:11:44.326 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_1 2024-03-25T18:11:44.382 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:11:44.382 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:11:44.382 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T18:11:44.382 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:11:44.382 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:11:44.383 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-03-25 16:22:14.836904048 +0000 2024-03-25T18:11:44.383 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-03-25 16:18:08.985114585 +0000 2024-03-25T18:11:44.383 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-03-25 16:18:08.985114585 +0000 2024-03-25T18:11:44.383 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-03-25 16:18:08.985114585 +0000 2024-03-25T18:11:44.383 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:11:44.449 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-03-25T18:11:44.449 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-03-25T18:11:44.449 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000213981 s, 2.4 MB/s 2024-03-25T18:11:44.450 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:11:44.506 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_2 2024-03-25T18:11:44.561 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:11:44.561 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:11:44.561 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T18:11:44.561 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:11:44.562 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:11:44.562 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-03-25 16:22:14.837904029 +0000 2024-03-25T18:11:44.562 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-03-25 16:18:09.331107885 +0000 2024-03-25T18:11:44.562 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-03-25 16:18:09.331107885 +0000 2024-03-25T18:11:44.562 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-03-25 16:18:09.331107885 +0000 2024-03-25T18:11:44.562 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:11:44.627 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-03-25T18:11:44.627 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-03-25T18:11:44.627 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000239482 s, 2.1 MB/s 2024-03-25T18:11:44.628 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:11:44.684 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_3 2024-03-25T18:11:44.739 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:11:44.739 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:11:44.739 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T18:11:44.739 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:11:44.739 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:11:44.739 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-03-25 16:22:14.837904029 +0000 2024-03-25T18:11:44.739 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-03-25 16:18:09.673101263 +0000 2024-03-25T18:11:44.740 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-03-25 16:18:09.673101263 +0000 2024-03-25T18:11:44.740 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-03-25 16:18:09.673101263 +0000 2024-03-25T18:11:44.740 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:11:44.804 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-03-25T18:11:44.805 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-03-25T18:11:44.805 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000230882 s, 2.2 MB/s 2024-03-25T18:11:44.806 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:11:44.860 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_4 2024-03-25T18:11:44.915 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:11:44.915 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:11:44.915 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T18:11:44.915 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:11:44.915 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:11:44.915 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-03-25 16:22:14.838904011 +0000 2024-03-25T18:11:44.915 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-03-25 16:18:10.004094853 +0000 2024-03-25T18:11:44.916 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-03-25 16:18:10.004094853 +0000 2024-03-25T18:11:44.916 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-03-25 16:18:10.004094853 +0000 2024-03-25T18:11:44.916 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:11:44.980 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-03-25T18:11:44.981 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-03-25T18:11:44.981 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000177916 s, 2.9 MB/s 2024-03-25T18:11:44.982 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:11:45.037 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:11:45.037 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:11:45.037 INFO:tasks.ceph:Generating config... 2024-03-25T18:11:45.038 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:11:45.038 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:11:45.038 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:11:45.038 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:11:45.038 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:11:45.038 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:11:45.038 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:11:45.039 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:11:45.039 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:11:45.039 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:11:45.039 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:11:45.039 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:11:45.039 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:11:45.039 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:11:45.039 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:11:45.039 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:11:45.039 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:11:45.039 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:11:45.039 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:11:45.039 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:11:45.039 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:11:45.039 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:11:45.040 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:11:45.040 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:11:45.040 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:11:45.040 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:11:45.040 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:11:45.040 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:11:45.040 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:11:45.040 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:11:45.040 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:11:45.040 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:11:45.040 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:11:45.040 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:11:45.040 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:11:45.040 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:11:45.040 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:11:45.041 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:11:45.041 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:11:45.041 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:11:45.041 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:11:45.041 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:11:45.041 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:11:45.041 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:11:45.041 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:11:45.041 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:11:45.041 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:11:45.041 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:11:45.041 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:11:45.041 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:11:45.041 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:11:45.042 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:11:45.042 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:11:45.042 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:11:45.042 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:11:45.122 INFO:teuthology.orchestra.run.smithi063.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:11:45.125 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:11:45.170 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:11:45.197 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.63:6789]'), ('mon.c', '[v1:172.21.15.63:6790]'), ('mon.b', '[v1:172.21.15.81:6789]')] 2024-03-25T18:11:45.197 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.63:6789],[v1:172.21.15.63:6790],[v1:172.21.15.81:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:11:45.198 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:11:45.198 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:11:45.251 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.63:6789]' --addv c '[v1:172.21.15.63:6790]' --addv b '[v1:172.21.15.81:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:11:45.336 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:11:45.390 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: generated fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:11:45.390 INFO:teuthology.orchestra.run.smithi063.stdout:setting min_mon_release = quincy 2024-03-25T18:11:45.391 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 0 2024-03-25T18:11:45.391 INFO:teuthology.orchestra.run.smithi063.stdout:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:11:45.391 INFO:teuthology.orchestra.run.smithi063.stdout:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:11:45.391 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:11:45.391 INFO:teuthology.orchestra.run.smithi063.stdout:min_mon_release 17 (quincy) 2024-03-25T18:11:45.391 INFO:teuthology.orchestra.run.smithi063.stdout:election_strategy: 3 2024-03-25T18:11:45.391 INFO:teuthology.orchestra.run.smithi063.stdout:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:11:45.391 INFO:teuthology.orchestra.run.smithi063.stdout:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:11:45.391 INFO:teuthology.orchestra.run.smithi063.stdout:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:11:45.391 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:11:45.393 DEBUG:teuthology.orchestra.run.smithi063:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:11:45.408 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9f4ce699-6031-4b74-90db-a99103627708... 2024-03-25T18:11:45.409 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:11:45.451 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:11:45.498 INFO:teuthology.orchestra.run.smithi063.stdout:[global] 2024-03-25T18:11:45.498 INFO:teuthology.orchestra.run.smithi063.stdout: chdir = "" 2024-03-25T18:11:45.498 INFO:teuthology.orchestra.run.smithi063.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:11:45.498 INFO:teuthology.orchestra.run.smithi063.stdout: auth supported = cephx 2024-03-25T18:11:45.498 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.498 INFO:teuthology.orchestra.run.smithi063.stdout: filestore xattr use omap = true 2024-03-25T18:11:45.499 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.499 INFO:teuthology.orchestra.run.smithi063.stdout: mon clock drift allowed = 1.000 2024-03-25T18:11:45.499 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.499 INFO:teuthology.orchestra.run.smithi063.stdout: osd crush chooseleaf type = 0 2024-03-25T18:11:45.499 INFO:teuthology.orchestra.run.smithi063.stdout: auth debug = true 2024-03-25T18:11:45.499 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.499 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on old message = true 2024-03-25T18:11:45.499 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on bug = true 2024-03-25T18:11:45.499 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.499 INFO:teuthology.orchestra.run.smithi063.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:11:45.499 INFO:teuthology.orchestra.run.smithi063.stdout: mon pg warn max object skew = 0 2024-03-25T18:11:45.499 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.500 INFO:teuthology.orchestra.run.smithi063.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:11:45.500 INFO:teuthology.orchestra.run.smithi063.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:11:45.500 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.500 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default size = 2 2024-03-25T18:11:45.500 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.500 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow primary affinity = true 2024-03-25T18:11:45.500 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow pg remap = true 2024-03-25T18:11:45.500 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:11:45.500 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:11:45.500 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on no sortbitwise = false 2024-03-25T18:11:45.500 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:11:45.500 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on too few osds = false 2024-03-25T18:11:45.500 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:11:45.500 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:11:45.501 INFO:teuthology.orchestra.run.smithi063.stdout: mon_allow_pool_size_one = true 2024-03-25T18:11:45.501 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.501 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:11:45.501 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.501 INFO:teuthology.orchestra.run.smithi063.stdout: osd default data pool replay window = 5 2024-03-25T18:11:45.501 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.501 INFO:teuthology.orchestra.run.smithi063.stdout: mon allow pool delete = true 2024-03-25T18:11:45.501 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.501 INFO:teuthology.orchestra.run.smithi063.stdout: mon cluster log file level = debug 2024-03-25T18:11:45.501 INFO:teuthology.orchestra.run.smithi063.stdout: debug asserts on shutdown = true 2024-03-25T18:11:45.501 INFO:teuthology.orchestra.run.smithi063.stdout: mon health detail to clog = false 2024-03-25T18:11:45.501 INFO:teuthology.orchestra.run.smithi063.stdout: mon host = "[v1:172.21.15.63:6789],[v1:172.21.15.63:6790],[v1:172.21.15.81:6789]" 2024-03-25T18:11:45.501 INFO:teuthology.orchestra.run.smithi063.stdout: mon client directed command retry = 5 2024-03-25T18:11:45.501 INFO:teuthology.orchestra.run.smithi063.stdout: mon election default strategy = 3 2024-03-25T18:11:45.501 INFO:teuthology.orchestra.run.smithi063.stdout: ms bind msgr2 = False 2024-03-25T18:11:45.502 INFO:teuthology.orchestra.run.smithi063.stdout: ms inject socket failures = 5000 2024-03-25T18:11:45.502 INFO:teuthology.orchestra.run.smithi063.stdout: ms type = async 2024-03-25T18:11:45.502 INFO:teuthology.orchestra.run.smithi063.stdout: osd heartbeat grace = 80 2024-03-25T18:11:45.502 INFO:teuthology.orchestra.run.smithi063.stdout: fsid = 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:11:45.502 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.502 INFO:teuthology.orchestra.run.smithi063.stdout:[osd] 2024-03-25T18:11:45.502 INFO:teuthology.orchestra.run.smithi063.stdout: osd journal size = 100 2024-03-25T18:11:45.502 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.502 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub load threshold = 5.0 2024-03-25T18:11:45.502 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub max interval = 600 2024-03-25T18:11:45.502 INFO:teuthology.orchestra.run.smithi063.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:11:45.502 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.502 INFO:teuthology.orchestra.run.smithi063.stdout: osd recover clone overlap = true 2024-03-25T18:11:45.502 INFO:teuthology.orchestra.run.smithi063.stdout: osd recovery max chunk = 1048576 2024-03-25T18:11:45.503 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.503 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug shutdown = true 2024-03-25T18:11:45.503 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug op order = true 2024-03-25T18:11:45.503 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify stray on activate = true 2024-03-25T18:11:45.503 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug trim objects = true 2024-03-25T18:11:45.503 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.503 INFO:teuthology.orchestra.run.smithi063.stdout: osd open classes on start = true 2024-03-25T18:11:45.503 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug pg log writeout = true 2024-03-25T18:11:45.503 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.503 INFO:teuthology.orchestra.run.smithi063.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:11:45.503 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.503 INFO:teuthology.orchestra.run.smithi063.stdout: osd map max advance = 10 2024-03-25T18:11:45.503 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.503 INFO:teuthology.orchestra.run.smithi063.stdout: journal zero on create = true 2024-03-25T18:11:45.503 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.504 INFO:teuthology.orchestra.run.smithi063.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:11:45.504 INFO:teuthology.orchestra.run.smithi063.stdout: filestore apply finisher threads = 3 2024-03-25T18:11:45.504 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.504 INFO:teuthology.orchestra.run.smithi063.stdout: bdev debug aio = true 2024-03-25T18:11:45.504 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug misdirected ops = true 2024-03-25T18:11:45.504 INFO:teuthology.orchestra.run.smithi063.stdout: bdev async discard = True 2024-03-25T18:11:45.504 INFO:teuthology.orchestra.run.smithi063.stdout: bdev enable discard = True 2024-03-25T18:11:45.504 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore allocator = stupid 2024-03-25T18:11:45.504 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore block size = 96636764160 2024-03-25T18:11:45.504 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore fsck on mount = True 2024-03-25T18:11:45.504 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore zero block detection = True 2024-03-25T18:11:45.504 INFO:teuthology.orchestra.run.smithi063.stdout: debug bluefs = 1 2024-03-25T18:11:45.504 INFO:teuthology.orchestra.run.smithi063.stdout: debug bluestore = 1 2024-03-25T18:11:45.504 INFO:teuthology.orchestra.run.smithi063.stdout: debug monc = 20 2024-03-25T18:11:45.505 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-03-25T18:11:45.505 INFO:teuthology.orchestra.run.smithi063.stdout: debug osd = 20 2024-03-25T18:11:45.505 INFO:teuthology.orchestra.run.smithi063.stdout: debug rocksdb = 10 2024-03-25T18:11:45.505 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:11:45.505 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd full ratio = 0.9 2024-03-25T18:11:45.505 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:11:45.505 INFO:teuthology.orchestra.run.smithi063.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:11:45.505 INFO:teuthology.orchestra.run.smithi063.stdout: osd class default list = * 2024-03-25T18:11:45.505 INFO:teuthology.orchestra.run.smithi063.stdout: osd class load list = * 2024-03-25T18:11:45.505 INFO:teuthology.orchestra.run.smithi063.stdout: osd client watch timeout = 120 2024-03-25T18:11:45.505 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify cached snaps = True 2024-03-25T18:11:45.505 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify missing on start = True 2024-03-25T18:11:45.505 INFO:teuthology.orchestra.run.smithi063.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:11:45.505 INFO:teuthology.orchestra.run.smithi063.stdout: osd fast shutdown = False 2024-03-25T18:11:45.505 INFO:teuthology.orchestra.run.smithi063.stdout: osd max markdown count = 1000 2024-03-25T18:11:45.506 INFO:teuthology.orchestra.run.smithi063.stdout: osd mclock override recovery settings = True 2024-03-25T18:11:45.506 INFO:teuthology.orchestra.run.smithi063.stdout: osd objectstore = bluestore 2024-03-25T18:11:45.506 INFO:teuthology.orchestra.run.smithi063.stdout: osd op queue = debug_random 2024-03-25T18:11:45.506 INFO:teuthology.orchestra.run.smithi063.stdout: osd op queue cut off = debug_random 2024-03-25T18:11:45.506 INFO:teuthology.orchestra.run.smithi063.stdout: osd shutdown pgref assert = True 2024-03-25T18:11:45.506 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.506 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr] 2024-03-25T18:11:45.506 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-03-25T18:11:45.506 INFO:teuthology.orchestra.run.smithi063.stdout: debug mgr = 20 2024-03-25T18:11:45.506 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2024-03-25T18:11:45.506 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2024-03-25T18:11:45.506 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:11:45.506 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:11:45.506 INFO:teuthology.orchestra.run.smithi063.stdout: mgr/telemetry/nag = false 2024-03-25T18:11:45.506 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.507 INFO:teuthology.orchestra.run.smithi063.stdout:[mon] 2024-03-25T18:11:45.507 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-03-25T18:11:45.507 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2024-03-25T18:11:45.507 INFO:teuthology.orchestra.run.smithi063.stdout: debug paxos = 20 2024-03-25T18:11:45.507 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2024-03-25T18:11:45.507 INFO:teuthology.orchestra.run.smithi063.stdout: mon data avail warn = 5 2024-03-25T18:11:45.507 INFO:teuthology.orchestra.run.smithi063.stdout: mon mgr mkfs grace = 240 2024-03-25T18:11:45.507 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:11:45.507 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd reporter subtree level = osd 2024-03-25T18:11:45.507 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd prime pg temp = true 2024-03-25T18:11:45.507 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:11:45.507 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.507 INFO:teuthology.orchestra.run.smithi063.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:11:45.507 INFO:teuthology.orchestra.run.smithi063.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:11:45.507 INFO:teuthology.orchestra.run.smithi063.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:11:45.508 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.508 INFO:teuthology.orchestra.run.smithi063.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:11:45.508 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:11:45.508 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:11:45.508 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.508 INFO:teuthology.orchestra.run.smithi063.stdout: # 1m isn't quite enough 2024-03-25T18:11:45.508 INFO:teuthology.orchestra.run.smithi063.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:11:45.508 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.508 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:11:45.508 INFO:teuthology.orchestra.run.smithi063.stdout: mon min osdmap epochs = 50 2024-03-25T18:11:45.508 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd crush smoke test = False 2024-03-25T18:11:45.508 INFO:teuthology.orchestra.run.smithi063.stdout: mon osdmap full prune interval = 2 2024-03-25T18:11:45.508 INFO:teuthology.orchestra.run.smithi063.stdout: mon osdmap full prune min = 15 2024-03-25T18:11:45.508 INFO:teuthology.orchestra.run.smithi063.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:11:45.508 INFO:teuthology.orchestra.run.smithi063.stdout: mon scrub interval = 300 2024-03-25T18:11:45.509 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on pool no app = False 2024-03-25T18:11:45.509 INFO:teuthology.orchestra.run.smithi063.stdout: paxos service trim min = 10 2024-03-25T18:11:45.509 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:11:45.509 INFO:teuthology.orchestra.run.smithi063.stdout:[client] 2024-03-25T18:11:45.509 INFO:teuthology.orchestra.run.smithi063.stdout: rgw cache enabled = true 2024-03-25T18:11:45.509 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable ops log = true 2024-03-25T18:11:45.509 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable usage log = true 2024-03-25T18:11:45.509 INFO:teuthology.orchestra.run.smithi063.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:11:45.509 INFO:teuthology.orchestra.run.smithi063.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:11:45.509 INFO:teuthology.orchestra.run.smithi063.stdout: debug monc = 20 2024-03-25T18:11:45.509 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-03-25T18:11:45.509 INFO:teuthology.orchestra.run.smithi063.stdout: debug objecter = 20 2024-03-25T18:11:45.509 INFO:teuthology.orchestra.run.smithi063.stdout: debug rados = 20 2024-03-25T18:11:45.509 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.a] 2024-03-25T18:11:45.509 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.c] 2024-03-25T18:11:45.510 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.b] 2024-03-25T18:11:45.510 INFO:teuthology.orchestra.run.smithi081.stdout:[global] 2024-03-25T18:11:45.510 INFO:teuthology.orchestra.run.smithi081.stdout: chdir = "" 2024-03-25T18:11:45.510 INFO:teuthology.orchestra.run.smithi081.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:11:45.510 INFO:teuthology.orchestra.run.smithi081.stdout: auth supported = cephx 2024-03-25T18:11:45.510 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.510 INFO:teuthology.orchestra.run.smithi081.stdout: filestore xattr use omap = true 2024-03-25T18:11:45.510 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.510 INFO:teuthology.orchestra.run.smithi081.stdout: mon clock drift allowed = 1.000 2024-03-25T18:11:45.510 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.511 INFO:teuthology.orchestra.run.smithi081.stdout: osd crush chooseleaf type = 0 2024-03-25T18:11:45.511 INFO:teuthology.orchestra.run.smithi081.stdout: auth debug = true 2024-03-25T18:11:45.511 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.511 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on old message = true 2024-03-25T18:11:45.511 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on bug = true 2024-03-25T18:11:45.511 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.511 INFO:teuthology.orchestra.run.smithi081.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:11:45.511 INFO:teuthology.orchestra.run.smithi081.stdout: mon pg warn max object skew = 0 2024-03-25T18:11:45.511 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.511 INFO:teuthology.orchestra.run.smithi081.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:11:45.511 INFO:teuthology.orchestra.run.smithi081.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:11:45.511 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.511 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default size = 2 2024-03-25T18:11:45.511 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.511 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow primary affinity = true 2024-03-25T18:11:45.511 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow pg remap = true 2024-03-25T18:11:45.512 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:11:45.512 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:11:45.512 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on no sortbitwise = false 2024-03-25T18:11:45.512 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:11:45.512 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on too few osds = false 2024-03-25T18:11:45.512 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:11:45.512 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:11:45.512 INFO:teuthology.orchestra.run.smithi081.stdout: mon_allow_pool_size_one = true 2024-03-25T18:11:45.512 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.512 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:11:45.512 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.512 INFO:teuthology.orchestra.run.smithi081.stdout: osd default data pool replay window = 5 2024-03-25T18:11:45.512 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.512 INFO:teuthology.orchestra.run.smithi081.stdout: mon allow pool delete = true 2024-03-25T18:11:45.512 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.512 INFO:teuthology.orchestra.run.smithi081.stdout: mon cluster log file level = debug 2024-03-25T18:11:45.513 INFO:teuthology.orchestra.run.smithi081.stdout: debug asserts on shutdown = true 2024-03-25T18:11:45.513 INFO:teuthology.orchestra.run.smithi081.stdout: mon health detail to clog = false 2024-03-25T18:11:45.513 INFO:teuthology.orchestra.run.smithi081.stdout: mon host = "[v1:172.21.15.63:6789],[v1:172.21.15.63:6790],[v1:172.21.15.81:6789]" 2024-03-25T18:11:45.513 INFO:teuthology.orchestra.run.smithi081.stdout: mon client directed command retry = 5 2024-03-25T18:11:45.513 INFO:teuthology.orchestra.run.smithi081.stdout: mon election default strategy = 3 2024-03-25T18:11:45.513 INFO:teuthology.orchestra.run.smithi081.stdout: ms bind msgr2 = False 2024-03-25T18:11:45.513 INFO:teuthology.orchestra.run.smithi081.stdout: ms inject socket failures = 5000 2024-03-25T18:11:45.513 INFO:teuthology.orchestra.run.smithi081.stdout: ms type = async 2024-03-25T18:11:45.513 INFO:teuthology.orchestra.run.smithi081.stdout: osd heartbeat grace = 80 2024-03-25T18:11:45.513 INFO:teuthology.orchestra.run.smithi081.stdout: fsid = 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:11:45.513 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.513 INFO:teuthology.orchestra.run.smithi081.stdout:[osd] 2024-03-25T18:11:45.513 INFO:teuthology.orchestra.run.smithi081.stdout: osd journal size = 100 2024-03-25T18:11:45.513 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.513 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub load threshold = 5.0 2024-03-25T18:11:45.513 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub max interval = 600 2024-03-25T18:11:45.514 INFO:teuthology.orchestra.run.smithi081.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:11:45.514 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.514 INFO:teuthology.orchestra.run.smithi081.stdout: osd recover clone overlap = true 2024-03-25T18:11:45.514 INFO:teuthology.orchestra.run.smithi081.stdout: osd recovery max chunk = 1048576 2024-03-25T18:11:45.514 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.514 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug shutdown = true 2024-03-25T18:11:45.514 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug op order = true 2024-03-25T18:11:45.514 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify stray on activate = true 2024-03-25T18:11:45.514 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug trim objects = true 2024-03-25T18:11:45.514 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.514 INFO:teuthology.orchestra.run.smithi081.stdout: osd open classes on start = true 2024-03-25T18:11:45.514 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug pg log writeout = true 2024-03-25T18:11:45.514 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.514 INFO:teuthology.orchestra.run.smithi081.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:11:45.514 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.515 INFO:teuthology.orchestra.run.smithi081.stdout: osd map max advance = 10 2024-03-25T18:11:45.515 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.515 INFO:teuthology.orchestra.run.smithi081.stdout: journal zero on create = true 2024-03-25T18:11:45.515 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.515 INFO:teuthology.orchestra.run.smithi081.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:11:45.515 INFO:teuthology.orchestra.run.smithi081.stdout: filestore apply finisher threads = 3 2024-03-25T18:11:45.515 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.515 INFO:teuthology.orchestra.run.smithi081.stdout: bdev debug aio = true 2024-03-25T18:11:45.515 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug misdirected ops = true 2024-03-25T18:11:45.515 INFO:teuthology.orchestra.run.smithi081.stdout: bdev async discard = True 2024-03-25T18:11:45.515 INFO:teuthology.orchestra.run.smithi081.stdout: bdev enable discard = True 2024-03-25T18:11:45.515 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore allocator = stupid 2024-03-25T18:11:45.515 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore block size = 96636764160 2024-03-25T18:11:45.515 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore fsck on mount = True 2024-03-25T18:11:45.515 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore zero block detection = True 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluefs = 1 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluestore = 1 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: debug monc = 20 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: debug osd = 20 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: debug rocksdb = 10 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd full ratio = 0.9 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: osd class default list = * 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: osd class load list = * 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: osd client watch timeout = 120 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify cached snaps = True 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify missing on start = True 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:11:45.516 INFO:teuthology.orchestra.run.smithi081.stdout: osd fast shutdown = False 2024-03-25T18:11:45.517 INFO:teuthology.orchestra.run.smithi081.stdout: osd max markdown count = 1000 2024-03-25T18:11:45.517 INFO:teuthology.orchestra.run.smithi081.stdout: osd mclock override recovery settings = True 2024-03-25T18:11:45.517 INFO:teuthology.orchestra.run.smithi081.stdout: osd objectstore = bluestore 2024-03-25T18:11:45.517 INFO:teuthology.orchestra.run.smithi081.stdout: osd op queue = debug_random 2024-03-25T18:11:45.517 INFO:teuthology.orchestra.run.smithi081.stdout: osd op queue cut off = debug_random 2024-03-25T18:11:45.517 INFO:teuthology.orchestra.run.smithi081.stdout: osd shutdown pgref assert = True 2024-03-25T18:11:45.517 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.517 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr] 2024-03-25T18:11:45.517 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-03-25T18:11:45.517 INFO:teuthology.orchestra.run.smithi081.stdout: debug mgr = 20 2024-03-25T18:11:45.517 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2024-03-25T18:11:45.517 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2024-03-25T18:11:45.517 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:11:45.518 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:11:45.518 INFO:teuthology.orchestra.run.smithi081.stdout: mgr/telemetry/nag = false 2024-03-25T18:11:45.518 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.518 INFO:teuthology.orchestra.run.smithi081.stdout:[mon] 2024-03-25T18:11:45.518 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-03-25T18:11:45.518 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2024-03-25T18:11:45.518 INFO:teuthology.orchestra.run.smithi081.stdout: debug paxos = 20 2024-03-25T18:11:45.518 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2024-03-25T18:11:45.518 INFO:teuthology.orchestra.run.smithi081.stdout: mon data avail warn = 5 2024-03-25T18:11:45.518 INFO:teuthology.orchestra.run.smithi081.stdout: mon mgr mkfs grace = 240 2024-03-25T18:11:45.518 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:11:45.518 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd reporter subtree level = osd 2024-03-25T18:11:45.518 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd prime pg temp = true 2024-03-25T18:11:45.518 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:11:45.519 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.519 INFO:teuthology.orchestra.run.smithi081.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:11:45.519 INFO:teuthology.orchestra.run.smithi081.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:11:45.519 INFO:teuthology.orchestra.run.smithi081.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:11:45.519 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.519 INFO:teuthology.orchestra.run.smithi081.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:11:45.519 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:11:45.519 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:11:45.519 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.519 INFO:teuthology.orchestra.run.smithi081.stdout: # 1m isn't quite enough 2024-03-25T18:11:45.519 INFO:teuthology.orchestra.run.smithi081.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:11:45.519 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.519 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:11:45.519 INFO:teuthology.orchestra.run.smithi081.stdout: mon min osdmap epochs = 50 2024-03-25T18:11:45.519 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd crush smoke test = False 2024-03-25T18:11:45.519 INFO:teuthology.orchestra.run.smithi081.stdout: mon osdmap full prune interval = 2 2024-03-25T18:11:45.520 INFO:teuthology.orchestra.run.smithi081.stdout: mon osdmap full prune min = 15 2024-03-25T18:11:45.520 INFO:teuthology.orchestra.run.smithi081.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:11:45.520 INFO:teuthology.orchestra.run.smithi081.stdout: mon scrub interval = 300 2024-03-25T18:11:45.520 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on pool no app = False 2024-03-25T18:11:45.520 INFO:teuthology.orchestra.run.smithi081.stdout: paxos service trim min = 10 2024-03-25T18:11:45.520 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:11:45.520 INFO:teuthology.orchestra.run.smithi081.stdout:[client] 2024-03-25T18:11:45.520 INFO:teuthology.orchestra.run.smithi081.stdout: rgw cache enabled = true 2024-03-25T18:11:45.520 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable ops log = true 2024-03-25T18:11:45.520 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable usage log = true 2024-03-25T18:11:45.520 INFO:teuthology.orchestra.run.smithi081.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:11:45.520 INFO:teuthology.orchestra.run.smithi081.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:11:45.520 INFO:teuthology.orchestra.run.smithi081.stdout: debug monc = 20 2024-03-25T18:11:45.521 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-03-25T18:11:45.521 INFO:teuthology.orchestra.run.smithi081.stdout: debug objecter = 20 2024-03-25T18:11:45.521 INFO:teuthology.orchestra.run.smithi081.stdout: debug rados = 20 2024-03-25T18:11:45.521 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.a] 2024-03-25T18:11:45.521 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.c] 2024-03-25T18:11:45.521 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.b] 2024-03-25T18:11:45.522 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:11:45.522 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:11:45.577 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:11:45.577 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:11:45.577 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:11:45.591 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:11:45.592 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:11:45.645 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T18:11:45.646 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:11:45.646 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:11:45.646 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:11:45.724 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:11:45.724 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:11:45.778 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T18:11:45.778 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:11:45.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:11:45.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:11:45.818 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:11:45.818 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:11:45.873 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:11:45.873 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:11:45.874 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:11:45.927 INFO:teuthology.orchestra.run.smithi063.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:11:45.930 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:11:45.994 INFO:teuthology.orchestra.run.smithi081.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:11:45.996 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:11:45.996 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:11:45.996 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:11:46.038 INFO:teuthology.orchestra.run.smithi063.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:11:46.053 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:11:46.095 INFO:teuthology.orchestra.run.smithi081.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:11:46.110 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:11:46.110 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:11:46.110 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:11:46.136 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:11:46.136 INFO:tasks.ceph:role: osd.0 2024-03-25T18:11:46.136 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T18:11:46.137 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:11:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:11:46.210 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:11:46.210 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:11:46.210 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:11:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:11:46.210 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:11:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:11:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:11:46.211 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:11:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:11:46.725 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-03-25T18:11:46.728 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-03-25T18:11:46.728 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:11:46.881 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:11:46.915 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:11:46.983 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:11:46.983 INFO:tasks.ceph:role: osd.1 2024-03-25T18:11:46.983 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T18:11:46.983 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:11:47.057 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:11:47.057 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:11:47.058 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:11:47.058 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:11:47.058 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:11:47.058 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:11:47.058 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:11:47.058 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:11:47.058 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:11:47.058 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:11:47.503 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-03-25T18:11:47.505 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-03-25T18:11:47.505 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:11:47.605 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:11:47.637 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:11:47.703 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:11:47.703 INFO:tasks.ceph:role: osd.2 2024-03-25T18:11:47.703 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T18:11:47.703 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:11:47.778 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:11:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:11:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:11:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:11:47.778 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:11:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:11:47.778 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:11:47.778 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:11:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:11:47.779 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:11:48.262 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-03-25T18:11:48.264 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-03-25T18:11:48.264 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:11:48.384 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:11:48.418 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:11:48.487 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:11:48.487 INFO:tasks.ceph:role: osd.3 2024-03-25T18:11:48.487 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T18:11:48.487 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:11:48.563 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:11:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:11:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:11:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:11:48.563 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:11:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:11:48.563 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:11:48.563 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:11:48.564 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:11:48.564 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:11:49.089 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-03-25T18:11:49.091 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-03-25T18:11:49.091 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:11:49.226 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:11:49.260 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:11:49.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:49.386+0000 7fb4df69d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:11:49.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:49.386+0000 7fb4df69d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:11:49.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:49.386+0000 7fb4df69d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:11:49.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:49.386+0000 7fb4df69d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:11:49.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:49.386+0000 7fb4df69d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:11:49.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:49.387+0000 7fb4df69d640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:11:52.236 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:11:52.267 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:11:52.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:52.358+0000 7f91e67fd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:11:52.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:52.359+0000 7f91e67fd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:11:52.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:52.359+0000 7f91e67fd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:11:52.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:52.359+0000 7f91e67fd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:11:52.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:52.359+0000 7f91e67fd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:11:52.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:52.359+0000 7f91e67fd640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:11:55.202 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:11:55.233 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:11:55.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:55.323+0000 7f952821a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:11:55.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:55.323+0000 7f952821a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:11:55.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:55.323+0000 7f952821a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:11:55.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:55.323+0000 7f952821a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:11:55.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:55.323+0000 7f952821a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:11:55.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:55.324+0000 7f952821a640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:11:58.164 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:11:58.195 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:11:58.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:58.285+0000 7fecf48a2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:11:58.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:58.285+0000 7fecf48a2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:11:58.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:58.285+0000 7fecf48a2640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:11:58.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:58.285+0000 7fecf48a2640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:11:58.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:58.285+0000 7fecf48a2640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:11:58.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:11:58.286+0000 7fecf48a2640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:12:01.137 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:12:01.168 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:12:01.199 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:12:01.200 INFO:tasks.ceph:role: osd.4 2024-03-25T18:12:01.200 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T18:12:01.200 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:12:01.276 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:12:01.276 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:12:01.276 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:12:01.276 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:12:01.276 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:12:01.276 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:12:01.276 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:12:01.276 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:12:01.276 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:12:01.276 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:12:01.789 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-03-25T18:12:01.791 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-03-25T18:12:01.791 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:12:01.937 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:12:01.974 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:12:02.045 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:12:02.045 INFO:tasks.ceph:role: osd.5 2024-03-25T18:12:02.045 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T18:12:02.045 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:12:02.120 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:12:02.120 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:12:02.120 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:12:02.121 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:12:02.121 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:12:02.121 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:12:02.121 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:12:02.121 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:12:02.121 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:12:02.121 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:12:02.566 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-03-25T18:12:02.568 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-03-25T18:12:02.568 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:12:02.651 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:12:02.722 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:12:02.789 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:12:02.789 INFO:tasks.ceph:role: osd.6 2024-03-25T18:12:02.789 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T18:12:02.789 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:12:02.863 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:12:02.863 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:12:02.864 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:12:02.864 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:12:02.864 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:12:02.864 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:12:02.864 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:12:02.864 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:12:02.864 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:12:02.864 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:12:03.355 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-03-25T18:12:03.357 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-03-25T18:12:03.357 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:12:03.479 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:12:03.509 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:12:03.576 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:12:03.576 INFO:tasks.ceph:role: osd.7 2024-03-25T18:12:03.576 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T18:12:03.576 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:12:03.651 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:12:03.651 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:12:03.651 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:12:03.651 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:12:03.651 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:12:03.651 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:12:03.651 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:12:03.651 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:12:03.651 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:12:03.651 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:12:04.162 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-03-25T18:12:04.163 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-03-25T18:12:04.164 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:12:04.289 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:12:04.322 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:12:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:04.417+0000 7feedefbb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:12:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:04.418+0000 7feedefbb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:12:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:04.418+0000 7feedefbb640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:12:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:04.418+0000 7feedefbb640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:12:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:04.418+0000 7feedefbb640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:12:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:04.418+0000 7feedefbb640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:12:07.267 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:12:07.299 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:12:07.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:07.389+0000 7fc3d8a10640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:12:07.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:07.389+0000 7fc3d8a10640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:12:07.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:07.389+0000 7fc3d8a10640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:12:07.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:07.389+0000 7fc3d8a10640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:12:07.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:07.389+0000 7fc3d8a10640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:12:07.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:07.389+0000 7fc3d8a10640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:12:10.245 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:12:10.275 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:12:10.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:10.364+0000 7fc852d91640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:12:10.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:10.364+0000 7fc852d91640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:12:10.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:10.364+0000 7fc852d91640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:12:10.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:10.364+0000 7fc852d91640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:12:10.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:10.364+0000 7fc852d91640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:12:10.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:10.365+0000 7fc852d91640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:12:13.215 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:12:13.246 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:12:13.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:13.335+0000 7f9afda0e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:12:13.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:13.335+0000 7f9afda0e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:12:13.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:13.335+0000 7f9afda0e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:12:13.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:13.335+0000 7f9afda0e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:12:13.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:13.335+0000 7f9afda0e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:12:13.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:13.335+0000 7f9afda0e640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:12:16.187 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:12:16.217 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:12:16.218 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:12:16.218 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:12:16.250 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:12:16.250 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:12:16.321 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:12:16.321 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:12:16.391 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:12:16.391 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:12:16.459 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:12:16.459 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:12:16.527 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:12:16.527 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:12:16.560 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:12:16.560 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:12:16.629 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:12:16.629 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:12:16.698 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:12:16.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:12:16.767 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:12:16.767 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:12:16.835 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:12:16.835 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:12:16.852 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:12:16.852 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:12:16.891 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:12:16.891 DEBUG:teuthology.orchestra.run.smithi063:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:12:16.894 DEBUG:teuthology.orchestra.run.smithi081:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:12:16.934 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr.y] 2024-03-25T18:12:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBhvgFm45pPNxAAJldKgBqxBKlU8bN6paGxbA== 2024-03-25T18:12:16.935 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.0] 2024-03-25T18:12:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBlvgFm0AUXFxAA4RVgGHUCMCmLJ13CDXQ9rA== 2024-03-25T18:12:16.935 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.1] 2024-03-25T18:12:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBovgFm+9N2FRAA1E7gSXjtxlve3McCvNZk+w== 2024-03-25T18:12:16.935 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.2] 2024-03-25T18:12:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBrvgFm+11XExAAFSbH5+t7hg7XlrcbdFX47Q== 2024-03-25T18:12:16.935 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.3] 2024-03-25T18:12:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBuvgFmlXoVERAA+qsTSLhuQVG+gdGpju/9aw== 2024-03-25T18:12:16.935 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr.x] 2024-03-25T18:12:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBhvgFmsQlHOxAAKBh+BaSAdXWa/cUedNKsQw== 2024-03-25T18:12:16.935 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.4] 2024-03-25T18:12:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQB0vgFmRA/2GBAA2zSj+jUjOaNLTOgHmfXWog== 2024-03-25T18:12:16.935 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.5] 2024-03-25T18:12:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQB3vgFm3AY/FxAAhPPXW1sc058xXZJtY39f6Q== 2024-03-25T18:12:16.936 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.6] 2024-03-25T18:12:16.936 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQB6vgFm1j7JFRAA30Ny2gaVmmupZT7wMW0nnA== 2024-03-25T18:12:16.936 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.7] 2024-03-25T18:12:16.936 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQB9vgFm9foNFBAAUckrDNaGj1mFfdSnpX4FaQ== 2024-03-25T18:12:16.936 INFO:teuthology.orchestra.run.smithi063.stdout:[client.0] 2024-03-25T18:12:16.936 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBivgFmd/hXAhAAuhaL4c1nog4vxGSFuHlh/A== 2024-03-25T18:12:16.936 INFO:teuthology.orchestra.run.smithi063.stdout:[client.1] 2024-03-25T18:12:16.936 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBivgFmtCCzBRAAUmxc7LPv0sRbv3JGCdkGow== 2024-03-25T18:12:16.957 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.y] 2024-03-25T18:12:16.957 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBhvgFm45pPNxAAJldKgBqxBKlU8bN6paGxbA== 2024-03-25T18:12:16.958 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.0] 2024-03-25T18:12:16.958 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBlvgFm0AUXFxAA4RVgGHUCMCmLJ13CDXQ9rA== 2024-03-25T18:12:16.958 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.1] 2024-03-25T18:12:16.958 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBovgFm+9N2FRAA1E7gSXjtxlve3McCvNZk+w== 2024-03-25T18:12:16.958 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.2] 2024-03-25T18:12:16.958 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBrvgFm+11XExAAFSbH5+t7hg7XlrcbdFX47Q== 2024-03-25T18:12:16.958 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.3] 2024-03-25T18:12:16.958 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBuvgFmlXoVERAA+qsTSLhuQVG+gdGpju/9aw== 2024-03-25T18:12:16.958 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.x] 2024-03-25T18:12:16.958 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBhvgFmsQlHOxAAKBh+BaSAdXWa/cUedNKsQw== 2024-03-25T18:12:16.958 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.4] 2024-03-25T18:12:16.958 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQB0vgFmRA/2GBAA2zSj+jUjOaNLTOgHmfXWog== 2024-03-25T18:12:16.959 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.5] 2024-03-25T18:12:16.959 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQB3vgFm3AY/FxAAhPPXW1sc058xXZJtY39f6Q== 2024-03-25T18:12:16.959 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.6] 2024-03-25T18:12:16.959 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQB6vgFm1j7JFRAA30Ny2gaVmmupZT7wMW0nnA== 2024-03-25T18:12:16.959 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.7] 2024-03-25T18:12:16.959 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQB9vgFm9foNFBAAUckrDNaGj1mFfdSnpX4FaQ== 2024-03-25T18:12:16.959 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-03-25T18:12:16.959 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBivgFmd/hXAhAAuhaL4c1nog4vxGSFuHlh/A== 2024-03-25T18:12:16.959 INFO:teuthology.orchestra.run.smithi081.stdout:[client.1] 2024-03-25T18:12:16.959 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBivgFmtCCzBRAAUmxc7LPv0sRbv3JGCdkGow== 2024-03-25T18:12:16.960 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:12:16.980 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:12:17.053 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:12:17.056 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:12:17.108 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:12:17.110 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:12:17.160 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:12:17.162 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:12:17.214 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:12:17.216 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:12:17.267 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:12:17.269 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:12:17.321 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:12:17.323 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:12:17.374 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:12:17.376 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:12:17.428 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:12:17.430 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:12:17.481 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:12:17.484 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:12:17.535 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:12:17.537 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:12:17.591 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:12:17.593 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:12:17.648 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:12:17.648 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:12:17.674 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:12:17.771 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:12:17.797 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:12:17.864 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:12:17.961 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:12:17.987 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:12:18.015 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:12:18.113 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:12:18.140 DEBUG:teuthology.orchestra.run.smithi063:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:12:18.142 DEBUG:teuthology.orchestra.run.smithi081:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:12:18.195 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:12:18.195 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:12:18.195 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:12:18.195 DEBUG:teuthology.orchestra.run.smithi063:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:12:18.197 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:12:18.197 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:12:18.197 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:12:18.198 DEBUG:teuthology.orchestra.run.smithi063:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:12:18.199 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:12:18.200 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:12:18.200 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:12:18.200 DEBUG:teuthology.orchestra.run.smithi081:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:12:18.238 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:12:18.238 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:12:18.238 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:12:18.239 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:12:18.241 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:12:18.241 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:12:18.241 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:12:18.243 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:12:18.243 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:12:18.243 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:12:18.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.359+0000 7f0a8d5b6640 10 monclient: get_monmap_and_config 2024-03-25T18:12:18.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.359+0000 7f0a8d5b6640 10 monclient: build_initial_monmap 2024-03-25T18:12:18.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.359+0000 7f0a8d5b6640 10 monclient: monmap: 2024-03-25T18:12:18.360 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-03-25T18:12:18.360 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:18.360 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T18:12:18.360639+0000 2024-03-25T18:12:18.360 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T18:12:18.360639+0000 2024-03-25T18:12:18.360 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:18.360 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:18.360 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:18.360 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:12:18.361 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:12:18.361 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T18:12:18.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.360+0000 7f0a8d5b6640 1 Processor -- start 2024-03-25T18:12:18.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.360+0000 7f0a8d5b6640 1 -- start start 2024-03-25T18:12:18.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.360+0000 7f0a8d5b6640 10 monclient: init 2024-03-25T18:12:18.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.360+0000 7f0a8d5b6640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:18.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.360+0000 7f0a8d5b6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:18.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.360+0000 7f0a8d5b6640 10 monclient(hunting): picked mon.noname-a con 0x7f0a880b4700 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:18.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.360+0000 7f0a8d5b6640 10 monclient(hunting): picked mon.noname-c con 0x7f0a88142a20 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:18.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.360+0000 7f0a8d5b6640 10 monclient(hunting): picked mon.noname-b con 0x7f0a880a96b0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:18.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.360+0000 7f0a8d5b6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a880a9d60 con 0x7f0a880b4700 2024-03-25T18:12:18.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.360+0000 7f0a8d5b6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a8813ff10 con 0x7f0a88142a20 2024-03-25T18:12:18.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.360+0000 7f0a8d5b6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a880aa1f0 con 0x7f0a880a96b0 2024-03-25T18:12:18.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.360+0000 7f0a8d5b6640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:18.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.360+0000 7f0a8d5b6640 10 monclient(hunting): authenticate will time out at 7432.547057s 2024-03-25T18:12:18.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.360+0000 7f0a877fe640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a8813f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T18:12:18.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.360+0000 7f0a86ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a880b4700 legacy=0x7f0a88142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T18:12:18.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.360+0000 7f0a867fc640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 legacy=0x7f0a8813f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T18:12:18.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.561+0000 7f0a877fe640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a8813f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T18:12:18.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.561+0000 7f0a86ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a880b4700 legacy=0x7f0a88142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T18:12:18.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.561+0000 7f0a867fc640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 legacy=0x7f0a8813f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T18:12:18.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.962+0000 7f0a877fe640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a8813f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T18:12:18.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.962+0000 7f0a86ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a880b4700 legacy=0x7f0a88142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T18:12:18.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:18.962+0000 7f0a867fc640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 legacy=0x7f0a8813f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T18:12:19.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:19.763+0000 7f0a86ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a880b4700 legacy=0x7f0a88142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T18:12:19.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:19.763+0000 7f0a877fe640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a8813f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T18:12:19.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:19.763+0000 7f0a867fc640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 legacy=0x7f0a8813f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T18:12:21.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a84ff9640 10 monclient(hunting): tick 2024-03-25T18:12:21.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a84ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:12:21.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a84ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:12:21.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a84ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:12:21.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a84ff9640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a8813f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:12:21.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a84ff9640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 legacy=0x7f0a8813f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:12:21.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a84ff9640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a880b4700 legacy=0x7f0a88142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:12:21.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a84ff9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:21.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0a74003680 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:21.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0a880a96b0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:21.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0a88142a20 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:21.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a84ff9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a880b4ae0 con 0x7f0a74003680 2024-03-25T18:12:21.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a84ff9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a880a9d60 con 0x7f0a88142a20 2024-03-25T18:12:21.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a86ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a74003680 legacy=0x7f0a74005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T18:12:21.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a84ff9640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a8813ff10 con 0x7f0a880a96b0 2024-03-25T18:12:21.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a84ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:21.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a867fc640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a74009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T18:12:21.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.361+0000 7f0a877fe640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 legacy=0x7f0a7400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T18:12:21.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.562+0000 7f0a86ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a74003680 legacy=0x7f0a74005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T18:12:21.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.562+0000 7f0a867fc640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a74009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T18:12:21.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.563+0000 7f0a877fe640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 legacy=0x7f0a7400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T18:12:21.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.963+0000 7f0a86ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a74003680 legacy=0x7f0a74005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T18:12:21.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.963+0000 7f0a867fc640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a74009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T18:12:21.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:21.964+0000 7f0a877fe640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 legacy=0x7f0a7400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T18:12:22.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:22.764+0000 7f0a86ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a74003680 legacy=0x7f0a74005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T18:12:22.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:22.764+0000 7f0a867fc640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a74009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T18:12:22.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:22.765+0000 7f0a877fe640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 legacy=0x7f0a7400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T18:12:24.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.361+0000 7f0a84ff9640 10 monclient(hunting): tick 2024-03-25T18:12:24.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.361+0000 7f0a84ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:12:24.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.361+0000 7f0a84ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:12:24.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.361+0000 7f0a84ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:12:24.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.361+0000 7f0a84ff9640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a74009300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:12:24.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.362+0000 7f0a84ff9640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 legacy=0x7f0a7400cbc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:12:24.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.362+0000 7f0a84ff9640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a74003680 legacy=0x7f0a74005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:12:24.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.362+0000 7f0a84ff9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:24.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.362+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0a880a96b0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:24.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.362+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0a74003680 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:24.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.362+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0a88142a20 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:24.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.362+0000 7f0a84ff9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a880aa1f0 con 0x7f0a74003680 2024-03-25T18:12:24.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.362+0000 7f0a84ff9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a880b4ae0 con 0x7f0a88142a20 2024-03-25T18:12:24.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.362+0000 7f0a84ff9640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a880a9d60 con 0x7f0a880a96b0 2024-03-25T18:12:24.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.362+0000 7f0a84ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:24.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.362+0000 7f0a86ffd640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a74010430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T18:12:24.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.362+0000 7f0a867fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a74003680 legacy=0x7f0a7400a310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T18:12:24.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.362+0000 7f0a877fe640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 legacy=0x7f0a7400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T18:12:24.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.563+0000 7f0a867fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a74003680 legacy=0x7f0a7400a310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T18:12:24.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.563+0000 7f0a86ffd640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a74010430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T18:12:24.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.563+0000 7f0a877fe640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 legacy=0x7f0a7400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T18:12:24.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.964+0000 7f0a867fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a74003680 legacy=0x7f0a7400a310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T18:12:24.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.964+0000 7f0a86ffd640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a74010430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T18:12:24.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:24.964+0000 7f0a877fe640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 legacy=0x7f0a7400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T18:12:25.186 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:00:06.847 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:25.492 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:00:07.153 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:25.679 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:00:07.372 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:25.680 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:00:07.372 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:25.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:25.765+0000 7f0a86ffd640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a74010430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T18:12:25.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:25.765+0000 7f0a867fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a74003680 legacy=0x7f0a7400a310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T18:12:25.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:25.765+0000 7f0a877fe640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 legacy=0x7f0a7400cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T18:12:26.006 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:00:07.699 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:26.007 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:00:07.699 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:27.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.362+0000 7f0a84ff9640 10 monclient(hunting): tick 2024-03-25T18:12:27.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.362+0000 7f0a84ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:12:27.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.362+0000 7f0a84ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:12:27.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.362+0000 7f0a84ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:12:27.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.362+0000 7f0a84ff9640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a74010430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:12:27.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.362+0000 7f0a84ff9640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 legacy=0x7f0a7400cbc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:12:27.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.362+0000 7f0a84ff9640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a74003680 legacy=0x7f0a7400a310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:12:27.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.362+0000 7f0a84ff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:27.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.362+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0a880a96b0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:27.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.362+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0a88142a20 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:27.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.362+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0a74003680 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:27.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.362+0000 7f0a84ff9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a8813ff10 con 0x7f0a74003680 2024-03-25T18:12:27.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.362+0000 7f0a84ff9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a880aa1f0 con 0x7f0a88142a20 2024-03-25T18:12:27.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.362+0000 7f0a84ff9640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a880b4ae0 con 0x7f0a880a96b0 2024-03-25T18:12:27.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.362+0000 7f0a84ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:27.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.402+0000 7f0a867fc640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 0x7f0a74010be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:40260/0 (socket says 172.21.15.63:40260) 2024-03-25T18:12:27.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.402+0000 7f0a867fc640 1 -- 172.21.15.63:0/181666346 learned_addr learned my addr 172.21.15.63:0/181666346 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:12:27.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.758+0000 7f0a86ffd640 1 -- 172.21.15.63:0/181666346 >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a74010430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T18:12:27.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.758+0000 7f0a86ffd640 1 -- 172.21.15.63:0/181666346 >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a74010430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:12:27.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.758+0000 7f0a86ffd640 1 --1- 172.21.15.63:0/181666346 >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 0x7f0a74010430 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:12:27.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.766+0000 7f0a877fe640 1 -- 172.21.15.63:0/181666346 >> v1:172.21.15.63:6789/0 conn(0x7f0a74003680 legacy=0x7f0a7400cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-25T18:12:27.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.766+0000 7f0a877fe640 1 -- 172.21.15.63:0/181666346 >> v1:172.21.15.63:6789/0 conn(0x7f0a74003680 legacy=0x7f0a7400cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:12:27.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:27.766+0000 7f0a877fe640 1 --1- 172.21.15.63:0/181666346 >> v1:172.21.15.63:6789/0 conn(0x7f0a74003680 0x7f0a7400cbc0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:12:30.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:30.361+0000 7f0a84ff9640 10 monclient(hunting): tick 2024-03-25T18:12:30.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:30.361+0000 7f0a84ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:12:30.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:30.361+0000 7f0a84ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:12:30.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:30.361+0000 7f0a84ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:12:30.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:30.361+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 >> v1:172.21.15.63:6790/0 conn(0x7f0a880a96b0 legacy=0x7f0a74010430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:30.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:30.361+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 >> v1:172.21.15.81:6789/0 conn(0x7f0a88142a20 legacy=0x7f0a74010be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:30.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:30.361+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 >> v1:172.21.15.63:6789/0 conn(0x7f0a74003680 legacy=0x7f0a7400cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:30.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:30.361+0000 7f0a84ff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:30.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:30.361+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0a74005ae0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:30.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:30.361+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0a74012500 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:30.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:30.361+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0a74003680 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:30.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:30.361+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a880a9d60 con 0x7f0a74003680 2024-03-25T18:12:30.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:30.361+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a74010c30 con 0x7f0a74012500 2024-03-25T18:12:30.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:30.361+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a74010e60 con 0x7f0a74005ae0 2024-03-25T18:12:30.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:30.361+0000 7f0a84ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:33.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:33.361+0000 7f0a84ff9640 10 monclient(hunting): tick 2024-03-25T18:12:33.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:33.361+0000 7f0a84ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:12:33.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:33.361+0000 7f0a84ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:12:33.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:33.361+0000 7f0a84ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:12:33.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:33.361+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 >> v1:172.21.15.63:6790/0 conn(0x7f0a74005ae0 legacy=0x7f0a7400a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:33.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:33.361+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 >> v1:172.21.15.81:6789/0 conn(0x7f0a74012500 legacy=0x7f0a740148f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:33.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:33.361+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 >> v1:172.21.15.63:6789/0 conn(0x7f0a74003680 legacy=0x7f0a74010430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:33.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:33.361+0000 7f0a84ff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:33.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:33.361+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0a7400d2f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:33.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:33.361+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0a7400de20 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:33.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:33.362+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0a74017120 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:33.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:33.362+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a74011090 con 0x7f0a74017120 2024-03-25T18:12:33.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:33.362+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a74010480 con 0x7f0a7400de20 2024-03-25T18:12:33.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:33.362+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a740106b0 con 0x7f0a7400d2f0 2024-03-25T18:12:33.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:33.362+0000 7f0a84ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:36.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.362+0000 7f0a84ff9640 10 monclient(hunting): tick 2024-03-25T18:12:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.362+0000 7f0a84ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:12:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.362+0000 7f0a84ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:12:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.362+0000 7f0a84ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:12:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.362+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 >> v1:172.21.15.63:6790/0 conn(0x7f0a7400d2f0 legacy=0x7f0a7400d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.362+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 >> v1:172.21.15.81:6789/0 conn(0x7f0a7400de20 legacy=0x7f0a7400cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.362+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 >> v1:172.21.15.63:6789/0 conn(0x7f0a74017120 legacy=0x7f0a74015000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.362+0000 7f0a84ff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.362+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0a7400de20 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.362+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0a74017120 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.362+0000 7f0a84ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0a7400d2f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.362+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a740108e0 con 0x7f0a74017120 2024-03-25T18:12:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.362+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a74015000 con 0x7f0a7400de20 2024-03-25T18:12:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.362+0000 7f0a84ff9640 1 -- 172.21.15.63:0/181666346 --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a740151e0 con 0x7f0a7400d2f0 2024-03-25T18:12:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.362+0000 7f0a84ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:36.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.578+0000 7f0a85ffb640 1 -- 172.21.15.63:0/181666346 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303557385 0 0) 0x7f0a740151e0 con 0x7f0a7400d2f0 2024-03-25T18:12:36.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.578+0000 7f0a85ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:36.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.578+0000 7f0a85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:36.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.578+0000 7f0a85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:36.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.578+0000 7f0a85ffb640 10 monclient(hunting): my global_id is 4106 2024-03-25T18:12:36.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.578+0000 7f0a85ffb640 1 -- 172.21.15.63:0/181666346 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a70003520 con 0x7f0a7400d2f0 2024-03-25T18:12:36.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.607+0000 7f0a85ffb640 1 -- 172.21.15.63:0/181666346 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449644297 0 0) 0x7f0a740108e0 con 0x7f0a74017120 2024-03-25T18:12:36.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.607+0000 7f0a85ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:36.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.607+0000 7f0a85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:36.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.607+0000 7f0a85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:36.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.607+0000 7f0a85ffb640 10 monclient(hunting): my global_id is 4104 2024-03-25T18:12:36.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.607+0000 7f0a85ffb640 1 -- 172.21.15.63:0/181666346 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a740151e0 con 0x7f0a74017120 2024-03-25T18:12:36.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.624+0000 7f0a85ffb640 1 -- 172.21.15.63:0/181666346 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225834822 0 0) 0x7f0a74015000 con 0x7f0a7400de20 2024-03-25T18:12:36.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.624+0000 7f0a85ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:36.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.624+0000 7f0a85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:36.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.624+0000 7f0a85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:36.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.624+0000 7f0a85ffb640 10 monclient(hunting): my global_id is 4102 2024-03-25T18:12:36.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.624+0000 7f0a85ffb640 1 -- 172.21.15.63:0/181666346 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a740108e0 con 0x7f0a7400de20 2024-03-25T18:12:36.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a85ffb640 1 -- 172.21.15.63:0/181666346 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4266982401 0 0) 0x7f0a740151e0 con 0x7f0a74017120 2024-03-25T18:12:36.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a85ffb640 1 -- 172.21.15.63:0/181666346 >> v1:172.21.15.63:6790/0 conn(0x7f0a7400d2f0 legacy=0x7f0a7400d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:36.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a85ffb640 1 -- 172.21.15.63:0/181666346 >> v1:172.21.15.81:6789/0 conn(0x7f0a7400de20 legacy=0x7f0a740148f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:36.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a85ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:12:36.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a85ffb640 1 monclient: found mon.noname-a 2024-03-25T18:12:36.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:36.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a85ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:12:36.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a85ffb640 1 -- 172.21.15.63:0/181666346 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a74015410 con 0x7f0a74017120 2024-03-25T18:12:36.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a85ffb640 10 monclient: _finish_auth 0 2024-03-25T18:12:36.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a85ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:12:36.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:36.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a85ffb640 1 -- 172.21.15.63:0/181666346 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f0a7c004150 con 0x7f0a74017120 2024-03-25T18:12:36.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a8d5b6640 5 monclient: authenticate success, global_id 4104 2024-03-25T18:12:36.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a8d5b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:36.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:36.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a85ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:36.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a85ffb640 10 monclient: dump: 2024-03-25T18:12:36.698 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T18:12:36.698 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:36.698 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:36.698 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:36.698 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:36.698 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:36.698 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:36.698 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:36.699 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:36.699 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T18:12:36.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.695+0000 7f0a8d5b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:36.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.704+0000 7f0a85ffb640 1 -- 172.21.15.63:0/181666346 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a7c0042b0 con 0x7f0a74017120 2024-03-25T18:12:36.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.704+0000 7f0a85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:36.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.704+0000 7f0a85ffb640 1 -- 172.21.15.63:0/181666346 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f0a7c0046c0 con 0x7f0a74017120 2024-03-25T18:12:36.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.704+0000 7f0a8d5b6640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:36.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.704+0000 7f0a8d5b6640 4 set_mon_vals no callback set 2024-03-25T18:12:36.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.704+0000 7f0a85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:36.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.704+0000 7f0a85ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:36.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.704+0000 7f0a85ffb640 10 monclient: dump: 2024-03-25T18:12:36.706 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T18:12:36.706 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:36.706 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:36.706 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:36.706 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:36.706 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:36.706 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:36.706 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:36.706 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:36.706 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T18:12:36.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.705+0000 7f0a8d5b6640 10 monclient: shutdown 2024-03-25T18:12:36.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.705+0000 7f0a8d5b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:36.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.705+0000 7f0a8d5b6640 1 -- 172.21.15.63:0/181666346 >> v1:172.21.15.63:6789/0 conn(0x7f0a74017120 legacy=0x7f0a7400cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:36.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.705+0000 7f0a8d5b6640 1 -- 172.21.15.63:0/181666346 shutdown_connections 2024-03-25T18:12:36.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.705+0000 7f0a8d5b6640 1 -- 172.21.15.63:0/181666346 >> 172.21.15.63:0/181666346 conn(0x7f0a880668e0 msgr2=0x7f0a88066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:36.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.705+0000 7f0a8d5b6640 1 -- 172.21.15.63:0/181666346 shutdown_connections 2024-03-25T18:12:36.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.705+0000 7f0a8d5b6640 1 -- 172.21.15.63:0/181666346 wait complete. 2024-03-25T18:12:36.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.706+0000 7f0a8d5b6640 10 monclient: build_initial_monmap 2024-03-25T18:12:36.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.706+0000 7f0a8d5b6640 10 monclient: monmap: 2024-03-25T18:12:36.708 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-03-25T18:12:36.708 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:36.708 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-03-25T18:12:36.708 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-03-25T18:12:36.709 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:36.709 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-03-25T18:12:36.709 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:36.709 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:12:36.709 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:12:36.709 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T18:12:36.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.706+0000 7f0a8d5b6640 1 librados: starting msgr at 2024-03-25T18:12:36.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.706+0000 7f0a8d5b6640 1 librados: starting objecter 2024-03-25T18:12:36.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.707+0000 7f0a8d5b6640 1 Processor -- start 2024-03-25T18:12:36.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.707+0000 7f0a8d5b6640 1 -- start start 2024-03-25T18:12:36.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.707+0000 7f0a8d5b6640 1 librados: setting wanted keys 2024-03-25T18:12:36.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.707+0000 7f0a8d5b6640 1 librados: calling monclient init 2024-03-25T18:12:36.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.707+0000 7f0a8d5b6640 10 monclient: init 2024-03-25T18:12:36.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.708+0000 7f0a8d5b6640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:36.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.708+0000 7f0a8d5b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:36.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.708+0000 7f0a8d5b6640 10 monclient(hunting): picked mon.noname-c con 0x7f0a7400d2f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:36.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.708+0000 7f0a8d5b6640 10 monclient(hunting): picked mon.noname-a con 0x7f0a7400de20 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:36.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.708+0000 7f0a8d5b6640 10 monclient(hunting): picked mon.noname-b con 0x7f0a74017120 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:36.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.708+0000 7f0a8d5b6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a880af970 con 0x7f0a7400de20 2024-03-25T18:12:36.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.708+0000 7f0a8d5b6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a880b0890 con 0x7f0a74017120 2024-03-25T18:12:36.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.708+0000 7f0a8d5b6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a880b0b00 con 0x7f0a7400d2f0 2024-03-25T18:12:36.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.708+0000 7f0a8d5b6640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:36.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.708+0000 7f0a8d5b6640 10 monclient(hunting): authenticate will time out at 7450.894402s 2024-03-25T18:12:36.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.723+0000 7f0a867fc640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f0a7400de20 0x7f0a881d7dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56032/0 (socket says 172.21.15.63:56032) 2024-03-25T18:12:36.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.723+0000 7f0a867fc640 1 -- 172.21.15.63:0/423796748 learned_addr learned my addr 172.21.15.63:0/423796748 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:12:36.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.728+0000 7f0a6f7fe640 10 client.?.objecter ms_handle_connect 0x7f0a7400d2f0 2024-03-25T18:12:36.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.728+0000 7f0a6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:36.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.735+0000 7f0a6f7fe640 10 client.?.objecter ms_handle_connect 0x7f0a7400de20 2024-03-25T18:12:36.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.735+0000 7f0a6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:36.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.744+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167426606 0 0) 0x7f0a880af970 con 0x7f0a7400de20 2024-03-25T18:12:36.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.744+0000 7f0a6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:36.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.744+0000 7f0a6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:36.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.744+0000 7f0a6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:36.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.744+0000 7f0a6f7fe640 10 monclient(hunting): my global_id is 4113 2024-03-25T18:12:36.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.744+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a740151e0 con 0x7f0a7400de20 2024-03-25T18:12:36.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.760+0000 7f0a6f7fe640 10 client.?.objecter ms_handle_connect 0x7f0a74017120 2024-03-25T18:12:36.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.760+0000 7f0a6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:36.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.785+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837688338 0 0) 0x7f0a880b0890 con 0x7f0a74017120 2024-03-25T18:12:36.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.785+0000 7f0a6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:36.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.785+0000 7f0a6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:36.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.785+0000 7f0a6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:36.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.785+0000 7f0a6f7fe640 10 monclient(hunting): my global_id is 4108 2024-03-25T18:12:36.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.785+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a880af970 con 0x7f0a74017120 2024-03-25T18:12:36.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.799+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731803821 0 0) 0x7f0a880b0b00 con 0x7f0a7400d2f0 2024-03-25T18:12:36.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.799+0000 7f0a6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:36.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.799+0000 7f0a6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:36.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.799+0000 7f0a6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:36.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.799+0000 7f0a6f7fe640 10 monclient(hunting): my global_id is 4112 2024-03-25T18:12:36.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.799+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a880b0890 con 0x7f0a7400d2f0 2024-03-25T18:12:36.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.840+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2352857844 0 0) 0x7f0a740151e0 con 0x7f0a7400de20 2024-03-25T18:12:36.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.840+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a880b0b00 con 0x7f0a7400de20 2024-03-25T18:12:36.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.840+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f0a7c004600 con 0x7f0a7400de20 2024-03-25T18:12:36.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.840+0000 7f0a6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:36.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.840+0000 7f0a6f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:36.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.840+0000 7f0a6f7fe640 10 monclient(hunting): dump: 2024-03-25T18:12:36.842 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T18:12:36.842 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:36.842 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:36.842 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:36.842 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:36.842 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:36.842 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:36.842 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:36.842 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:36.842 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T18:12:36.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.840+0000 7f0a6f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:36.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1690091206 0 0) 0x7f0a880b0b00 con 0x7f0a7400de20 2024-03-25T18:12:36.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 >> v1:172.21.15.63:6790/0 conn(0x7f0a7400d2f0 legacy=0x7f0a88137f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:36.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 >> v1:172.21.15.81:6789/0 conn(0x7f0a74017120 legacy=0x7f0a881e3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:36.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a6f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:12:36.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a6f7fe640 1 monclient: found mon.a 2024-03-25T18:12:36.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:36.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a6f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:12:36.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a880b0de0 con 0x7f0a7400de20 2024-03-25T18:12:36.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a6f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:12:36.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a6f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:12:36.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:36.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a8d5b6640 5 monclient: authenticate success, global_id 4113 2024-03-25T18:12:36.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a8d5b6640 10 monclient: _renew_subs 2024-03-25T18:12:36.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a8d5b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:12:36.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a8d5b6640 1 -- 172.21.15.63:0/423796748 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a880ae9f0 con 0x7f0a7400de20 2024-03-25T18:12:36.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a8d5b6640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:36.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a8d5b6640 10 monclient: _renew_subs 2024-03-25T18:12:36.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a8d5b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:12:36.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a8d5b6640 1 -- 172.21.15.63:0/423796748 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a880af2c0 con 0x7f0a7400de20 2024-03-25T18:12:36.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.887+0000 7f0a8d5b6640 1 librados: init done 2024-03-25T18:12:36.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.888+0000 7f0a8d5b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:36.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.888+0000 7f0a8d5b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:36.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.888+0000 7f0a8d5b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:12:36.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:36.888+0000 7f0a8d5b6640 1 -- 172.21.15.63:0/423796748 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a880b8550 con 0x7f0a7400de20 2024-03-25T18:12:37.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.069+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a7c004a10 con 0x7f0a7400de20 2024-03-25T18:12:37.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.069+0000 7f0a6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:37.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.069+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f0a7c004e20 con 0x7f0a7400de20 2024-03-25T18:12:37.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.069+0000 7f0a6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:37.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.069+0000 7f0a6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:37.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.069+0000 7f0a6f7fe640 10 monclient: dump: 2024-03-25T18:12:37.071 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T18:12:37.071 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:37.071 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:37.071 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:37.071 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:37.071 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:37.071 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:37.071 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:37.071 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:37.071 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T18:12:37.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.069+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0a7c0053d0 con 0x7f0a7400de20 2024-03-25T18:12:37.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.069+0000 7f0a857fa640 4 set_mon_vals no callback set 2024-03-25T18:12:37.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.069+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 4239443402 0 0) 0x7f0a7c006490 con 0x7f0a7400de20 2024-03-25T18:12:37.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.069+0000 7f0a6f7fe640 10 client.4113.objecter ms_dispatch 0x7f0a88000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:12:37.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.069+0000 7f0a6f7fe640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:12:37.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.069+0000 7f0a6f7fe640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:12:37.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.069+0000 7f0a6f7fe640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-25T18:12:37.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.069+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0a7c006b20 con 0x7f0a7400de20 2024-03-25T18:12:37.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.069+0000 7f0a6f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:37.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.069+0000 7f0a6f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:37.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.178+0000 7f0a8d5b6640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:12:37.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.178+0000 7f0a8d5b6640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:12:37.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.178+0000 7f0a8d5b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:12:37.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.178+0000 7f0a8d5b6640 1 -- 172.21.15.63:0/423796748 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f0a880b8550 con 0x7f0a7400de20 2024-03-25T18:12:37.204 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:37.203+0000 7f0c0da1f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:12:37.258 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:37.257+0000 7f0c0da1f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:12:37.344 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:37.343+0000 7fd9c8010040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:12:37.403 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:37.403+0000 7fd9c8010040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:12:37.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.452+0000 7f0a6f7fe640 1 -- 172.21.15.63:0/423796748 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f0a7c005940 con 0x7f0a7400de20 2024-03-25T18:12:37.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.452+0000 7f0a6f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:12:37.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.452+0000 7f0a6f7fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:12:37.453 INFO:teuthology.orchestra.run.smithi063.stderr:adjusted tunables profile to default 2024-03-25T18:12:37.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.453+0000 7f0a8d5b6640 10 librados: watch_flush enter 2024-03-25T18:12:37.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.453+0000 7f0a8d5b6640 10 librados: watch_flush exit 2024-03-25T18:12:37.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.453+0000 7f0a8d5b6640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-25T18:12:37.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.453+0000 7f0a8d5b6640 10 client.4113.objecter successfully canceled tick 2024-03-25T18:12:37.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.453+0000 7f0a8d5b6640 10 monclient: shutdown 2024-03-25T18:12:37.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.453+0000 7f0a8d5b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:37.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.453+0000 7f0a8d5b6640 1 -- 172.21.15.63:0/423796748 >> v1:172.21.15.63:6789/0 conn(0x7f0a7400de20 legacy=0x7f0a881d7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:37.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.453+0000 7f0a8d5b6640 1 -- 172.21.15.63:0/423796748 shutdown_connections 2024-03-25T18:12:37.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.453+0000 7f0a8d5b6640 1 -- 172.21.15.63:0/423796748 >> 172.21.15.63:0/423796748 conn(0x7f0a880668e0 msgr2=0x7f0a880b7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:37.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.453+0000 7f0a8d5b6640 1 -- 172.21.15.63:0/423796748 shutdown_connections 2024-03-25T18:12:37.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.453+0000 7f0a8d5b6640 1 -- 172.21.15.63:0/423796748 wait complete. 2024-03-25T18:12:37.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:37.453+0000 7f0a8d5b6640 1 librados: shutdown 2024-03-25T18:12:37.466 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:12:37.466 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:12:37.466 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:12:37.466 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:12:37.494 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:12:37.494 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:12:37.560 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:12:37.560 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:12:37.627 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T18:12:37.627 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:12:37.693 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:12:37.694 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:12:37.720 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:12:37.720 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:12:37.785 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:37.784+0000 7f0c0da1f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:12:37.787 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:12:37.787 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:12:37.842 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:37.841+0000 7f0c0da1f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:12:37.855 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:12:37.855 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:12:37.888 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:37.887+0000 7f0c0da1f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:12:37.915 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 7d4153f5-c1a1-4a43-9781-52018a420e7e 0 2024-03-25T18:12:37.952 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:37.951+0000 7f0c0da1f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:12:37.983 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:37.983+0000 7fd9c8010040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:12:37.996 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:37.996+0000 7f0c0da1f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:12:38.039 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:38.039+0000 7fd9c8010040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:12:38.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.050+0000 7f37627cf640 10 monclient: get_monmap_and_config 2024-03-25T18:12:38.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.050+0000 7f37627cf640 10 monclient: build_initial_monmap 2024-03-25T18:12:38.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.050+0000 7f37627cf640 10 monclient: monmap: 2024-03-25T18:12:38.051 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T18:12:38.051 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:38.051 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:12:38.051019+0000 2024-03-25T18:12:38.051 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:12:38.051019+0000 2024-03-25T18:12:38.051 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:38.051 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:38.051 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:38.051 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:12:38.051 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:12:38.051 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:38.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.050+0000 7f37627cf640 1 Processor -- start 2024-03-25T18:12:38.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.050+0000 7f37627cf640 1 -- start start 2024-03-25T18:12:38.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.050+0000 7f37627cf640 10 monclient: init 2024-03-25T18:12:38.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.051+0000 7f37627cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:38.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.051+0000 7f37627cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:38.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.051+0000 7f37627cf640 10 monclient(hunting): picked mon.noname-a con 0x7f375c070210 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:38.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.051+0000 7f37627cf640 10 monclient(hunting): picked mon.noname-b con 0x7f375c0b06d0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:38.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.051+0000 7f37627cf640 10 monclient(hunting): picked mon.noname-c con 0x7f375c0705f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:38.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.051+0000 7f37627cf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375c0b0b80 con 0x7f375c070210 2024-03-25T18:12:38.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.051+0000 7f37627cf640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375c0b0430 con 0x7f375c0705f0 2024-03-25T18:12:38.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.051+0000 7f37627cf640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375c071670 con 0x7f375c0b06d0 2024-03-25T18:12:38.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.051+0000 7f37627cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:38.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.051+0000 7f37627cf640 10 monclient(hunting): authenticate will time out at 7463.231113s 2024-03-25T18:12:38.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.051+0000 7f375b7fe640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f375c0b06d0 0x7f375c0afd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.81:48164/0 (socket says 172.21.15.81:48164) 2024-03-25T18:12:38.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.051+0000 7f375b7fe640 1 -- 172.21.15.81:0/1509744996 learned_addr learned my addr 172.21.15.81:0/1509744996 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T18:12:38.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.054+0000 7f375affd640 1 -- 172.21.15.81:0/1509744996 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887825490 0 0) 0x7f375c071670 con 0x7f375c0b06d0 2024-03-25T18:12:38.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.054+0000 7f375affd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:38.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.054+0000 7f375affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:38.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.054+0000 7f375affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:38.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.054+0000 7f375affd640 10 monclient(hunting): my global_id is 4115 2024-03-25T18:12:38.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.054+0000 7f375affd640 1 -- 172.21.15.81:0/1509744996 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3744003520 con 0x7f375c0b06d0 2024-03-25T18:12:38.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.054+0000 7f375affd640 1 -- 172.21.15.81:0/1509744996 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930042965 0 0) 0x7f3754001490 con 0x7f375c0705f0 2024-03-25T18:12:38.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.054+0000 7f375affd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:38.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.054+0000 7f375affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:38.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.054+0000 7f375affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:38.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.054+0000 7f375affd640 10 monclient(hunting): my global_id is 4111 2024-03-25T18:12:38.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.054+0000 7f375affd640 1 -- 172.21.15.81:0/1509744996 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f375c071670 con 0x7f375c0705f0 2024-03-25T18:12:38.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.055+0000 7f375affd640 1 -- 172.21.15.81:0/1509744996 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324543069 0 0) 0x7f375c0b0b80 con 0x7f375c070210 2024-03-25T18:12:38.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.055+0000 7f375affd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:38.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.055+0000 7f375affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:38.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.055+0000 7f375affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:38.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.055+0000 7f375affd640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:12:38.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.055+0000 7f375affd640 1 -- 172.21.15.81:0/1509744996 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3754001490 con 0x7f375c070210 2024-03-25T18:12:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f375affd640 1 -- 172.21.15.81:0/1509744996 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1321914505 0 0) 0x7f3744003520 con 0x7f375c0b06d0 2024-03-25T18:12:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f375affd640 1 -- 172.21.15.81:0/1509744996 >> v1:172.21.15.81:6789/0 conn(0x7f375c0705f0 legacy=0x7f375c14b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f375affd640 1 -- 172.21.15.81:0/1509744996 >> v1:172.21.15.63:6789/0 conn(0x7f375c070210 legacy=0x7f375c140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f375affd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f375affd640 1 monclient: found mon.noname-b 2024-03-25T18:12:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f375affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f375affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:12:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f375affd640 1 -- 172.21.15.81:0/1509744996 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c070a60 con 0x7f375c0b06d0 2024-03-25T18:12:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f375affd640 10 monclient: _finish_auth 0 2024-03-25T18:12:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f375affd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f375affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f375affd640 1 -- 172.21.15.81:0/1509744996 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f374c001690 con 0x7f375c0b06d0 2024-03-25T18:12:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f375affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f375affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f375affd640 10 monclient: dump: 2024-03-25T18:12:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:38.061 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:38.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f37627cf640 5 monclient: authenticate success, global_id 4115 2024-03-25T18:12:38.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.058+0000 7f37627cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:38.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.085+0000 7f375affd640 1 -- 172.21.15.81:0/1509744996 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f374c0017f0 con 0x7f375c0b06d0 2024-03-25T18:12:38.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.085+0000 7f375affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:38.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.085+0000 7f375affd640 1 -- 172.21.15.81:0/1509744996 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f374c001b80 con 0x7f375c0b06d0 2024-03-25T18:12:38.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.085+0000 7f375affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:38.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.085+0000 7f375affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:38.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.085+0000 7f375affd640 10 monclient: dump: 2024-03-25T18:12:38.086 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:38.086 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:38.086 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.086 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.086 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:38.086 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:38.086 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:38.086 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:38.086 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:38.087 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:38.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.085+0000 7f37627cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:38.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.085+0000 7f37627cf640 4 set_mon_vals no callback set 2024-03-25T18:12:38.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.085+0000 7f37627cf640 10 monclient: shutdown 2024-03-25T18:12:38.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.085+0000 7f37627cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:38.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.085+0000 7f37627cf640 1 -- 172.21.15.81:0/1509744996 >> v1:172.21.15.63:6790/0 conn(0x7f375c0b06d0 legacy=0x7f375c0afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:38.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.085+0000 7f37627cf640 1 -- 172.21.15.81:0/1509744996 shutdown_connections 2024-03-25T18:12:38.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.085+0000 7f37627cf640 1 -- 172.21.15.81:0/1509744996 >> 172.21.15.81:0/1509744996 conn(0x7f375c0b5cd0 msgr2=0x7f375c0b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:38.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.085+0000 7f37627cf640 1 -- 172.21.15.81:0/1509744996 shutdown_connections 2024-03-25T18:12:38.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.085+0000 7f37627cf640 1 -- 172.21.15.81:0/1509744996 wait complete. 2024-03-25T18:12:38.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.086+0000 7f37627cf640 10 monclient: build_initial_monmap 2024-03-25T18:12:38.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.086+0000 7f37627cf640 10 monclient: monmap: 2024-03-25T18:12:38.090 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T18:12:38.090 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:38.090 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-25T18:12:38.090 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-25T18:12:38.090 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:38.090 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-25T18:12:38.090 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:38.090 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:12:38.090 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:12:38.090 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:38.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.086+0000 7f37627cf640 1 librados: starting msgr at 2024-03-25T18:12:38.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.086+0000 7f37627cf640 1 librados: starting objecter 2024-03-25T18:12:38.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.086+0000 7f37627cf640 1 Processor -- start 2024-03-25T18:12:38.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.087+0000 7f37627cf640 1 -- start start 2024-03-25T18:12:38.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.087+0000 7f37627cf640 1 librados: setting wanted keys 2024-03-25T18:12:38.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.087+0000 7f37627cf640 1 librados: calling monclient init 2024-03-25T18:12:38.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.087+0000 7f37627cf640 10 monclient: init 2024-03-25T18:12:38.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.087+0000 7f37627cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:38.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.087+0000 7f37627cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:38.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.087+0000 7f37627cf640 10 monclient(hunting): picked mon.noname-b con 0x7f375c070210 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:38.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.087+0000 7f37627cf640 10 monclient(hunting): picked mon.noname-a con 0x7f375c0705f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:38.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.087+0000 7f37627cf640 10 monclient(hunting): picked mon.noname-c con 0x7f375c0b06d0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:38.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.087+0000 7f37627cf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375c13adf0 con 0x7f375c0705f0 2024-03-25T18:12:38.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.087+0000 7f37627cf640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375c13b0d0 con 0x7f375c070210 2024-03-25T18:12:38.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.087+0000 7f37627cf640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375c13b3b0 con 0x7f375c0b06d0 2024-03-25T18:12:38.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.087+0000 7f37627cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:38.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.087+0000 7f37627cf640 10 monclient(hunting): authenticate will time out at 7463.267438s 2024-03-25T18:12:38.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.088+0000 7f3760d6c640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f375c0b06d0 0x7f375c13cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.81:48180/0 (socket says 172.21.15.81:48180) 2024-03-25T18:12:38.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.088+0000 7f3760d6c640 1 -- 172.21.15.81:0/3390588771 learned_addr learned my addr 172.21.15.81:0/3390588771 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T18:12:38.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.088+0000 7f3758ff9640 10 client.?.objecter ms_handle_connect 0x7f375c0b06d0 2024-03-25T18:12:38.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.088+0000 7f3758ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:38.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.088+0000 7f3758ff9640 10 client.?.objecter ms_handle_connect 0x7f375c070210 2024-03-25T18:12:38.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.088+0000 7f3758ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:38.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.088+0000 7f3758ff9640 10 client.?.objecter ms_handle_connect 0x7f375c0705f0 2024-03-25T18:12:38.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.088+0000 7f3758ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:38.093 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:38.087+0000 7fd9c8010040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:12:38.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.090+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747688709 0 0) 0x7f375c13b3b0 con 0x7f375c0b06d0 2024-03-25T18:12:38.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.090+0000 7f3758ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:38.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.090+0000 7f3758ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.090+0000 7f3758ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.090+0000 7f3758ff9640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:12:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.090+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3748004eb0 con 0x7f375c0b06d0 2024-03-25T18:12:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.091+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556782277 0 0) 0x7f375c13b0d0 con 0x7f375c070210 2024-03-25T18:12:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.091+0000 7f3758ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.091+0000 7f3758ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.091+0000 7f3758ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.091+0000 7f3758ff9640 10 monclient(hunting): my global_id is 4114 2024-03-25T18:12:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.091+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f375c13b3b0 con 0x7f375c070210 2024-03-25T18:12:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.091+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721806783 0 0) 0x7f375c13adf0 con 0x7f375c0705f0 2024-03-25T18:12:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.091+0000 7f3758ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.091+0000 7f3758ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.091+0000 7f3758ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:38.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.091+0000 7f3758ff9640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:12:38.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.091+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f375c13b0d0 con 0x7f375c0705f0 2024-03-25T18:12:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2602362362 0 0) 0x7f375c13b0d0 con 0x7f375c0705f0 2024-03-25T18:12:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f375c13adf0 con 0x7f375c0705f0 2024-03-25T18:12:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f374c001b00 con 0x7f375c0705f0 2024-03-25T18:12:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:38.098 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 523271719 0 0) 0x7f3748004eb0 con 0x7f375c0b06d0 2024-03-25T18:12:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f375c13b0d0 con 0x7f375c0b06d0 2024-03-25T18:12:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3754001d20 con 0x7f375c0b06d0 2024-03-25T18:12:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:38.099 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3536495361 0 0) 0x7f375c13b3b0 con 0x7f375c070210 2024-03-25T18:12:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3748004eb0 con 0x7f375c070210 2024-03-25T18:12:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3750001870 con 0x7f375c070210 2024-03-25T18:12:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.095+0000 7f3758ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:38.100 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:38.100 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.100 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.100 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:38.100 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:38.100 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:38.100 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:38.100 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:38.100 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:38.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2539633284 0 0) 0x7f375c13adf0 con 0x7f375c0705f0 2024-03-25T18:12:38.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 >> v1:172.21.15.63:6790/0 conn(0x7f375c0b06d0 legacy=0x7f375c13cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 >> v1:172.21.15.81:6789/0 conn(0x7f375c070210 legacy=0x7f375c142c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f3758ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:12:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f3758ff9640 1 monclient: found mon.a 2024-03-25T18:12:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f3758ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f3758ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:12:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c06d480 con 0x7f375c0705f0 2024-03-25T18:12:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f3758ff9640 10 monclient: _finish_auth 0 2024-03-25T18:12:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f3758ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:12:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f3758ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f37627cf640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:12:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f37627cf640 10 monclient: _renew_subs 2024-03-25T18:12:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f37627cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:12:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f37627cf640 1 -- 172.21.15.81:0/3390588771 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375c06d9a0 con 0x7f375c0705f0 2024-03-25T18:12:38.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f37627cf640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:38.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f37627cf640 10 monclient: _renew_subs 2024-03-25T18:12:38.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f37627cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:12:38.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f37627cf640 1 -- 172.21.15.81:0/3390588771 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f375c143580 con 0x7f375c0705f0 2024-03-25T18:12:38.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.099+0000 7f37627cf640 1 librados: init done 2024-03-25T18:12:38.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.100+0000 7f37627cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:38.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.100+0000 7f37627cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:38.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.100+0000 7f37627cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:12:38.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.100+0000 7f37627cf640 1 -- 172.21.15.81:0/3390588771 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f375c071200 con 0x7f375c0705f0 2024-03-25T18:12:38.120 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:38.120+0000 7f0c0da1f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:12:38.147 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:38.146+0000 7fd9c8010040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:12:38.189 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:38.189+0000 7f0c0da1f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:12:38.192 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:38.191+0000 7fd9c8010040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:12:38.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.205+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f374c0011e0 con 0x7f375c0705f0 2024-03-25T18:12:38.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.205+0000 7f3758ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:38.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.205+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f374c001c60 con 0x7f375c0705f0 2024-03-25T18:12:38.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.205+0000 7f3758ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:38.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.205+0000 7f3758ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:38.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.205+0000 7f3758ff9640 10 monclient: dump: 2024-03-25T18:12:38.206 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:38.206 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:38.207 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.207 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.207 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:38.207 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:38.207 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:38.207 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:38.207 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:38.207 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:38.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.205+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f374c004e20 con 0x7f375c0705f0 2024-03-25T18:12:38.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.205+0000 7f375a7fc640 4 set_mon_vals no callback set 2024-03-25T18:12:38.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.206+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2635367004 0 0) 0x7f374c006230 con 0x7f375c0705f0 2024-03-25T18:12:38.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.206+0000 7f3758ff9640 10 client.4119.objecter ms_dispatch 0x7f375c000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:12:38.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.206+0000 7f3758ff9640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:12:38.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.206+0000 7f3758ff9640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:12:38.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.206+0000 7f3758ff9640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T18:12:38.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.206+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f374c0052b0 con 0x7f375c0705f0 2024-03-25T18:12:38.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.206+0000 7f3758ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:38.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.206+0000 7f3758ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:38.236 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:38.235+0000 7f0c0da1f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:12:38.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.279+0000 7f37627cf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7d4153f5-c1a1-4a43-9781-52018a420e7e", "id": 0}] 2024-03-25T18:12:38.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.279+0000 7f37627cf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7d4153f5-c1a1-4a43-9781-52018a420e7e", "id": 0}] 2024-03-25T18:12:38.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.279+0000 7f37627cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:12:38.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.279+0000 7f37627cf640 1 -- 172.21.15.81:0/3390588771 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7d4153f5-c1a1-4a43-9781-52018a420e7e", "id": 0} v 0) v1 -- 0x7f375c04f730 con 0x7f375c0705f0 2024-03-25T18:12:38.290 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:38.289+0000 7f0c0da1f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:12:38.313 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:38.312+0000 7fd9c8010040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:12:38.337 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:38.336+0000 7f0c0da1f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:12:38.391 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:38.391+0000 7fd9c8010040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:12:38.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.393+0000 7f3758ff9640 1 -- 172.21.15.81:0/3390588771 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7d4153f5-c1a1-4a43-9781-52018a420e7e", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3713758288 0 732707515) 0x7f374c0055a0 con 0x7f375c0705f0 2024-03-25T18:12:38.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.393+0000 7f3758ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7d4153f5-c1a1-4a43-9781-52018a420e7e", "id": 0}] 2024-03-25T18:12:38.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.393+0000 7f3758ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:38.394 INFO:teuthology.orchestra.run.smithi081.stdout:0 2024-03-25T18:12:38.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.393+0000 7f37627cf640 10 librados: watch_flush enter 2024-03-25T18:12:38.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.394+0000 7f37627cf640 10 librados: watch_flush exit 2024-03-25T18:12:38.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.394+0000 7f37627cf640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T18:12:38.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.394+0000 7f37627cf640 10 client.4119.objecter successfully canceled tick 2024-03-25T18:12:38.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.394+0000 7f37627cf640 10 monclient: shutdown 2024-03-25T18:12:38.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.394+0000 7f37627cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:38.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.394+0000 7f37627cf640 1 -- 172.21.15.81:0/3390588771 >> v1:172.21.15.63:6789/0 conn(0x7f375c0705f0 legacy=0x7f375c13c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:38.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.394+0000 7f37627cf640 1 -- 172.21.15.81:0/3390588771 shutdown_connections 2024-03-25T18:12:38.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.394+0000 7f37627cf640 1 -- 172.21.15.81:0/3390588771 >> 172.21.15.81:0/3390588771 conn(0x7f375c0b5cd0 msgr2=0x7f375c13df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:38.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.394+0000 7f37627cf640 1 -- 172.21.15.81:0/3390588771 shutdown_connections 2024-03-25T18:12:38.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.394+0000 7f37627cf640 1 -- 172.21.15.81:0/3390588771 wait complete. 2024-03-25T18:12:38.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.394+0000 7f37627cf640 1 librados: shutdown 2024-03-25T18:12:38.405 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 6d5c570a-fd90-450a-a56c-f646b4d63b2c 1 2024-03-25T18:12:38.441 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:38.440+0000 7fd9c8010040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:12:38.497 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:38.496+0000 7fd9c8010040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:12:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.505+0000 7f2f627ed640 10 monclient: get_monmap_and_config 2024-03-25T18:12:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.505+0000 7f2f627ed640 10 monclient: build_initial_monmap 2024-03-25T18:12:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.505+0000 7f2f627ed640 10 monclient: monmap: 2024-03-25T18:12:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T18:12:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:12:38.506179+0000 2024-03-25T18:12:38.507 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:12:38.506179+0000 2024-03-25T18:12:38.507 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:38.507 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:38.507 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:38.507 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:12:38.507 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:12:38.507 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:38.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.505+0000 7f2f627ed640 1 Processor -- start 2024-03-25T18:12:38.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.505+0000 7f2f627ed640 1 -- start start 2024-03-25T18:12:38.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.505+0000 7f2f627ed640 10 monclient: init 2024-03-25T18:12:38.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.506+0000 7f2f627ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:38.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.506+0000 7f2f627ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:38.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.506+0000 7f2f627ed640 10 monclient(hunting): picked mon.noname-a con 0x7f2f5c0b4700 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:38.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.506+0000 7f2f627ed640 10 monclient(hunting): picked mon.noname-b con 0x7f2f5c142a20 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:38.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.506+0000 7f2f627ed640 10 monclient(hunting): picked mon.noname-c con 0x7f2f5c0a96b0 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:38.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.506+0000 7f2f627ed640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c0a9d60 con 0x7f2f5c0b4700 2024-03-25T18:12:38.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.506+0000 7f2f627ed640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c13ff10 con 0x7f2f5c0a96b0 2024-03-25T18:12:38.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.506+0000 7f2f627ed640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c0aa1f0 con 0x7f2f5c142a20 2024-03-25T18:12:38.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.506+0000 7f2f627ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:38.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.506+0000 7f2f627ed640 10 monclient(hunting): authenticate will time out at 7463.686189s 2024-03-25T18:12:38.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.507+0000 7f2f5b7fe640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f2f5c142a20 0x7f2f5c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.81:48188/0 (socket says 172.21.15.81:48188) 2024-03-25T18:12:38.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.507+0000 7f2f5b7fe640 1 -- 172.21.15.81:0/3379070680 learned_addr learned my addr 172.21.15.81:0/3379070680 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T18:12:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.509+0000 7f2f5affd640 1 -- 172.21.15.81:0/3379070680 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050374735 0 0) 0x7f2f5c13ff10 con 0x7f2f5c0a96b0 2024-03-25T18:12:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.509+0000 7f2f5affd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.509+0000 7f2f5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.509+0000 7f2f5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.509+0000 7f2f5affd640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:12:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.509+0000 7f2f5affd640 1 -- 172.21.15.81:0/3379070680 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f44003520 con 0x7f2f5c0a96b0 2024-03-25T18:12:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.509+0000 7f2f5affd640 1 -- 172.21.15.81:0/3379070680 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180173556 0 0) 0x7f2f5c0aa1f0 con 0x7f2f5c142a20 2024-03-25T18:12:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.509+0000 7f2f5affd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.509+0000 7f2f5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.509+0000 7f2f5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.509+0000 7f2f5affd640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:12:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.509+0000 7f2f5affd640 1 -- 172.21.15.81:0/3379070680 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c13ff10 con 0x7f2f5c142a20 2024-03-25T18:12:38.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f5affd640 1 -- 172.21.15.81:0/3379070680 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3390453724 0 0) 0x7f2f44003520 con 0x7f2f5c0a96b0 2024-03-25T18:12:38.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f5affd640 1 -- 172.21.15.81:0/3379070680 >> v1:172.21.15.63:6790/0 conn(0x7f2f5c142a20 legacy=0x7f2f5c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:38.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f5affd640 1 -- 172.21.15.81:0/3379070680 >> v1:172.21.15.63:6789/0 conn(0x7f2f5c0b4700 legacy=0x7f2f5c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:12:38.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f5affd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:38.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f5affd640 1 monclient: found mon.noname-c 2024-03-25T18:12:38.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:38.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f5affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:12:38.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f5affd640 1 -- 172.21.15.81:0/3379070680 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c0b4ae0 con 0x7f2f5c0a96b0 2024-03-25T18:12:38.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f5affd640 10 monclient: _finish_auth 0 2024-03-25T18:12:38.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f5affd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:38.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:38.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f5affd640 1 -- 172.21.15.81:0/3379070680 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2f54001a10 con 0x7f2f5c0a96b0 2024-03-25T18:12:38.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f627ed640 5 monclient: authenticate success, global_id 4117 2024-03-25T18:12:38.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f627ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:38.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:38.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f5affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:38.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f5affd640 10 monclient: dump: 2024-03-25T18:12:38.514 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:38.515 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:38.515 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.515 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.515 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:38.515 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:38.515 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:38.515 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:38.515 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:38.515 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:38.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.512+0000 7f2f627ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:38.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.547+0000 7f2f5affd640 1 -- 172.21.15.81:0/3379070680 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f54001b70 con 0x7f2f5c0a96b0 2024-03-25T18:12:38.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.547+0000 7f2f5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:38.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.547+0000 7f2f5affd640 1 -- 172.21.15.81:0/3379070680 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2f54001f80 con 0x7f2f5c0a96b0 2024-03-25T18:12:38.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.547+0000 7f2f5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:38.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.547+0000 7f2f5affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:38.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.547+0000 7f2f5affd640 10 monclient: dump: 2024-03-25T18:12:38.548 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:38.548 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:38.548 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.548 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.548 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:38.548 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:38.548 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:38.548 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:38.549 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:38.549 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:38.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.547+0000 7f2f627ed640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:38.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.547+0000 7f2f627ed640 4 set_mon_vals no callback set 2024-03-25T18:12:38.549 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:38.548+0000 7fd9c8010040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:12:38.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.547+0000 7f2f627ed640 10 monclient: shutdown 2024-03-25T18:12:38.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.547+0000 7f2f627ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:38.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.547+0000 7f2f627ed640 1 -- 172.21.15.81:0/3379070680 >> v1:172.21.15.81:6789/0 conn(0x7f2f5c0a96b0 legacy=0x7f2f5c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:38.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.548+0000 7f2f627ed640 1 -- 172.21.15.81:0/3379070680 shutdown_connections 2024-03-25T18:12:38.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.548+0000 7f2f627ed640 1 -- 172.21.15.81:0/3379070680 >> 172.21.15.81:0/3379070680 conn(0x7f2f5c0668e0 msgr2=0x7f2f5c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:38.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.548+0000 7f2f627ed640 1 -- 172.21.15.81:0/3379070680 shutdown_connections 2024-03-25T18:12:38.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.548+0000 7f2f627ed640 1 -- 172.21.15.81:0/3379070680 wait complete. 2024-03-25T18:12:38.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.548+0000 7f2f627ed640 10 monclient: build_initial_monmap 2024-03-25T18:12:38.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.548+0000 7f2f627ed640 10 monclient: monmap: 2024-03-25T18:12:38.552 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T18:12:38.552 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:38.552 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-25T18:12:38.552 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-25T18:12:38.552 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:38.552 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-25T18:12:38.552 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:38.552 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:12:38.552 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:12:38.553 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.549+0000 7f2f627ed640 1 librados: starting msgr at 2024-03-25T18:12:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.549+0000 7f2f627ed640 1 librados: starting objecter 2024-03-25T18:12:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.549+0000 7f2f627ed640 1 Processor -- start 2024-03-25T18:12:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.549+0000 7f2f627ed640 1 -- start start 2024-03-25T18:12:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.549+0000 7f2f627ed640 1 librados: setting wanted keys 2024-03-25T18:12:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.549+0000 7f2f627ed640 1 librados: calling monclient init 2024-03-25T18:12:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.549+0000 7f2f627ed640 10 monclient: init 2024-03-25T18:12:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.550+0000 7f2f627ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.550+0000 7f2f627ed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.550+0000 7f2f627ed640 10 monclient(hunting): picked mon.noname-a con 0x7f2f5c0a96b0 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.550+0000 7f2f627ed640 10 monclient(hunting): picked mon.noname-b con 0x7f2f5c0b4700 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.550+0000 7f2f627ed640 10 monclient(hunting): picked mon.noname-c con 0x7f2f5c142a20 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:38.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.550+0000 7f2f627ed640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c13b2c0 con 0x7f2f5c0a96b0 2024-03-25T18:12:38.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.550+0000 7f2f627ed640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c13a4b0 con 0x7f2f5c0b4700 2024-03-25T18:12:38.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.550+0000 7f2f627ed640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c13a790 con 0x7f2f5c142a20 2024-03-25T18:12:38.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.550+0000 7f2f627ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:38.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.550+0000 7f2f627ed640 10 monclient(hunting): authenticate will time out at 7463.729756s 2024-03-25T18:12:38.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.550+0000 7f2f60d8a640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f2f5c142a20 0x7f2f5c1e2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.81:48198/0 (socket says 172.21.15.81:48198) 2024-03-25T18:12:38.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.550+0000 7f2f60d8a640 1 -- 172.21.15.81:0/2493419236 learned_addr learned my addr 172.21.15.81:0/2493419236 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T18:12:38.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.551+0000 7f2f58ff9640 10 client.?.objecter ms_handle_connect 0x7f2f5c142a20 2024-03-25T18:12:38.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.551+0000 7f2f58ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:38.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.551+0000 7f2f58ff9640 10 client.?.objecter ms_handle_connect 0x7f2f5c0b4700 2024-03-25T18:12:38.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.551+0000 7f2f58ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:38.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.553+0000 7f2f58ff9640 10 client.?.objecter ms_handle_connect 0x7f2f5c0a96b0 2024-03-25T18:12:38.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.553+0000 7f2f58ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:38.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.586+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350337598 0 0) 0x7f2f5c13b2c0 con 0x7f2f5c0a96b0 2024-03-25T18:12:38.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.586+0000 7f2f58ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:38.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.586+0000 7f2f58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:38.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.586+0000 7f2f58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:38.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.586+0000 7f2f58ff9640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:12:38.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.586+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f48004eb0 con 0x7f2f5c0a96b0 2024-03-25T18:12:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.588+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596458786 0 0) 0x7f2f5c13a4b0 con 0x7f2f5c0b4700 2024-03-25T18:12:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.588+0000 7f2f58ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.588+0000 7f2f58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.588+0000 7f2f58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.588+0000 7f2f58ff9640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:12:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.588+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c13b2c0 con 0x7f2f5c0b4700 2024-03-25T18:12:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.588+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94958321 0 0) 0x7f2f5c13a790 con 0x7f2f5c142a20 2024-03-25T18:12:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.588+0000 7f2f58ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.588+0000 7f2f58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.588+0000 7f2f58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.588+0000 7f2f58ff9640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:12:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.588+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c13a4b0 con 0x7f2f5c142a20 2024-03-25T18:12:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.593+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 811497890 0 0) 0x7f2f5c13a4b0 con 0x7f2f5c142a20 2024-03-25T18:12:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.593+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f5c13a790 con 0x7f2f5c142a20 2024-03-25T18:12:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.593+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3279314775 0 0) 0x7f2f5c13b2c0 con 0x7f2f5c0b4700 2024-03-25T18:12:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.593+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f5c13a4b0 con 0x7f2f5c0b4700 2024-03-25T18:12:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.593+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2f54001ec0 con 0x7f2f5c142a20 2024-03-25T18:12:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.593+0000 7f2f58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.593+0000 7f2f58ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.593+0000 7f2f58ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.593+0000 7f2f58ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.593+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2f500018b0 con 0x7f2f5c0b4700 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.593+0000 7f2f58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.593+0000 7f2f58ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.593+0000 7f2f58ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:38.597 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:38.597 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:38.597 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:38.597 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:38.597 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:38.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.594+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2729038100 0 0) 0x7f2f48004eb0 con 0x7f2f5c0a96b0 2024-03-25T18:12:38.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.594+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f5c13b2c0 con 0x7f2f5c0a96b0 2024-03-25T18:12:38.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.594+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2f4c001510 con 0x7f2f5c0a96b0 2024-03-25T18:12:38.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.594+0000 7f2f58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:38.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.594+0000 7f2f58ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:38.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.594+0000 7f2f58ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:38.597 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:38.597 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:38.597 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.597 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.598 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:38.598 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:38.598 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:38.598 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:38.598 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:38.598 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:38.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2005786648 0 0) 0x7f2f5c13a790 con 0x7f2f5c142a20 2024-03-25T18:12:38.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 >> v1:172.21.15.81:6789/0 conn(0x7f2f5c0b4700 legacy=0x7f2f5c13abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:38.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 >> v1:172.21.15.63:6789/0 conn(0x7f2f5c0a96b0 legacy=0x7f2f5c1391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:38.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f58ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:12:38.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f58ff9640 1 monclient: found mon.c 2024-03-25T18:12:38.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f58ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f58ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c138cb0 con 0x7f2f5c142a20 2024-03-25T18:12:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f58ff9640 10 monclient: _finish_auth 0 2024-03-25T18:12:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f58ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:12:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f627ed640 5 monclient: authenticate success, global_id 4124 2024-03-25T18:12:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f627ed640 10 monclient: _renew_subs 2024-03-25T18:12:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f627ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f627ed640 1 -- 172.21.15.81:0/2493419236 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f5c1389b0 con 0x7f2f5c142a20 2024-03-25T18:12:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f627ed640 10 client.4124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f627ed640 10 monclient: _renew_subs 2024-03-25T18:12:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f627ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f627ed640 1 -- 172.21.15.81:0/2493419236 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f5c1357d0 con 0x7f2f5c142a20 2024-03-25T18:12:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.597+0000 7f2f627ed640 1 librados: init done 2024-03-25T18:12:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.598+0000 7f2f627ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.598+0000 7f2f627ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:38.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.598+0000 7f2f627ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:38.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.598+0000 7f2f627ed640 1 -- 172.21.15.81:0/2493419236 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f5c0aa5f0 con 0x7f2f5c142a20 2024-03-25T18:12:38.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.621+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f54003c20 con 0x7f2f5c142a20 2024-03-25T18:12:38.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.621+0000 7f2f58ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:38.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.621+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2f54003ff0 con 0x7f2f5c142a20 2024-03-25T18:12:38.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.621+0000 7f2f58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:38.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.621+0000 7f2f58ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:38.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.621+0000 7f2f58ff9640 10 monclient: dump: 2024-03-25T18:12:38.622 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:38.622 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:38.622 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.622 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:38.623 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:38.623 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:38.623 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:38.623 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:38.623 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:38.623 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:38.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.621+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2f540045a0 con 0x7f2f5c142a20 2024-03-25T18:12:38.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.621+0000 7f2f5a7fc640 4 set_mon_vals no callback set 2024-03-25T18:12:38.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.621+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1367565312 0 0) 0x7f2f540068b0 con 0x7f2f5c142a20 2024-03-25T18:12:38.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.621+0000 7f2f58ff9640 10 client.4124.objecter ms_dispatch 0x7f2f5c000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:12:38.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.621+0000 7f2f58ff9640 3 client.4124.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:12:38.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.621+0000 7f2f58ff9640 3 client.4124.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:12:38.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.621+0000 7f2f58ff9640 20 client.4124.objecter dump_active .. 0 homeless 2024-03-25T18:12:38.731 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:38.730+0000 7f0c0da1f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:12:38.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.798+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2f54005b00 con 0x7f2f5c142a20 2024-03-25T18:12:38.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.798+0000 7f2f58ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:38.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.798+0000 7f2f58ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:38.866 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:38.865+0000 7f0c0da1f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:12:38.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.869+0000 7f2f627ed640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6d5c570a-fd90-450a-a56c-f646b4d63b2c", "id": 1}] 2024-03-25T18:12:38.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.869+0000 7f2f627ed640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6d5c570a-fd90-450a-a56c-f646b4d63b2c", "id": 1}] 2024-03-25T18:12:38.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.869+0000 7f2f627ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:38.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:38.869+0000 7f2f627ed640 1 -- 172.21.15.81:0/2493419236 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd new", "uuid": "6d5c570a-fd90-450a-a56c-f646b4d63b2c", "id": 1} v 0) v1 -- 0x7f2f5c135b20 con 0x7f2f5c142a20 2024-03-25T18:12:38.958 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:38.958+0000 7fd9c8010040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:12:38.992 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:38.992+0000 7f0c0da1f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:12:39.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.011+0000 7f2f58ff9640 1 -- 172.21.15.81:0/2493419236 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6d5c570a-fd90-450a-a56c-f646b4d63b2c", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3958506460 0 940483276) 0x7f2f54005cf0 con 0x7f2f5c142a20 2024-03-25T18:12:39.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.011+0000 7f2f58ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6d5c570a-fd90-450a-a56c-f646b4d63b2c", "id": 1}] 2024-03-25T18:12:39.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.011+0000 7f2f58ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:39.011 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-03-25T18:12:39.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.011+0000 7f2f627ed640 10 librados: watch_flush enter 2024-03-25T18:12:39.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.011+0000 7f2f627ed640 10 librados: watch_flush exit 2024-03-25T18:12:39.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.011+0000 7f2f627ed640 20 client.4124.objecter shutdown clearing up homeless session... 2024-03-25T18:12:39.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.011+0000 7f2f627ed640 10 client.4124.objecter successfully canceled tick 2024-03-25T18:12:39.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.011+0000 7f2f627ed640 10 monclient: shutdown 2024-03-25T18:12:39.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.011+0000 7f2f627ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:39.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.011+0000 7f2f627ed640 1 -- 172.21.15.81:0/2493419236 >> v1:172.21.15.63:6790/0 conn(0x7f2f5c142a20 legacy=0x7f2f5c1e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:39.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.011+0000 7f2f627ed640 1 -- 172.21.15.81:0/2493419236 shutdown_connections 2024-03-25T18:12:39.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.011+0000 7f2f627ed640 1 -- 172.21.15.81:0/2493419236 >> 172.21.15.81:0/2493419236 conn(0x7f2f5c0668e0 msgr2=0x7f2f5c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:39.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.012+0000 7f2f627ed640 1 -- 172.21.15.81:0/2493419236 shutdown_connections 2024-03-25T18:12:39.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.012+0000 7f2f627ed640 1 -- 172.21.15.81:0/2493419236 wait complete. 2024-03-25T18:12:39.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.012+0000 7f2f627ed640 1 librados: shutdown 2024-03-25T18:12:39.023 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new cdfc79b2-cb55-4ddb-bfac-682073ac86dc 2 2024-03-25T18:12:39.041 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:39.040+0000 7f0c0da1f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:12:39.101 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:39.100+0000 7fd9c8010040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:12:39.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.122+0000 7f3e566d3640 10 monclient: get_monmap_and_config 2024-03-25T18:12:39.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.122+0000 7f3e566d3640 10 monclient: build_initial_monmap 2024-03-25T18:12:39.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.122+0000 7f3e566d3640 10 monclient: monmap: 2024-03-25T18:12:39.122 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T18:12:39.123 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:39.123 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:12:39.123072+0000 2024-03-25T18:12:39.123 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:12:39.123072+0000 2024-03-25T18:12:39.123 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:39.123 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:39.123 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:39.123 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:12:39.123 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:12:39.123 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:39.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.122+0000 7f3e566d3640 1 Processor -- start 2024-03-25T18:12:39.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.122+0000 7f3e566d3640 1 -- start start 2024-03-25T18:12:39.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.122+0000 7f3e566d3640 10 monclient: init 2024-03-25T18:12:39.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.123+0000 7f3e566d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:39.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.123+0000 7f3e566d3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:39.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.123+0000 7f3e566d3640 10 monclient(hunting): picked mon.noname-a con 0x7f3e500a9920 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:39.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.123+0000 7f3e566d3640 10 monclient(hunting): picked mon.noname-b con 0x7f3e500b45a0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:39.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.123+0000 7f3e566d3640 10 monclient(hunting): picked mon.noname-c con 0x7f3e50146450 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:39.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.123+0000 7f3e566d3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e5006b100 con 0x7f3e500a9920 2024-03-25T18:12:39.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.123+0000 7f3e566d3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e500b2940 con 0x7f3e50146450 2024-03-25T18:12:39.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.123+0000 7f3e566d3640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e5013fec0 con 0x7f3e500b45a0 2024-03-25T18:12:39.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.123+0000 7f3e566d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:39.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.123+0000 7f3e566d3640 10 monclient(hunting): authenticate will time out at 7464.303110s 2024-03-25T18:12:39.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.124+0000 7f3e4f7fe640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f3e500b45a0 0x7f3e50145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.81:48208/0 (socket says 172.21.15.81:48208) 2024-03-25T18:12:39.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.124+0000 7f3e4f7fe640 1 -- 172.21.15.81:0/1700190030 learned_addr learned my addr 172.21.15.81:0/1700190030 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T18:12:39.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.124+0000 7f3e54c70640 1 --1- 172.21.15.81:0/1700190030 >> v1:172.21.15.81:6789/0 conn(0x7f3e50146450 0x7f3e5014dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:47290/0 (socket says 172.21.15.81:47290) 2024-03-25T18:12:39.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.126+0000 7f3e4effd640 1 -- 172.21.15.81:0/1700190030 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4358265 0 0) 0x7f3e500b2940 con 0x7f3e50146450 2024-03-25T18:12:39.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.126+0000 7f3e4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:39.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.126+0000 7f3e4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:39.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.126+0000 7f3e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:39.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.126+0000 7f3e4effd640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:12:39.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.126+0000 7f3e4effd640 1 -- 172.21.15.81:0/1700190030 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e38003520 con 0x7f3e50146450 2024-03-25T18:12:39.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.126+0000 7f3e4effd640 1 -- 172.21.15.81:0/1700190030 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209553254 0 0) 0x7f3e5013fec0 con 0x7f3e500b45a0 2024-03-25T18:12:39.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.126+0000 7f3e4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:39.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.126+0000 7f3e4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:39.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.126+0000 7f3e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:39.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.126+0000 7f3e4effd640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:12:39.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.126+0000 7f3e4effd640 1 -- 172.21.15.81:0/1700190030 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e500b2940 con 0x7f3e500b45a0 2024-03-25T18:12:39.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.127+0000 7f3e4effd640 1 -- 172.21.15.81:0/1700190030 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457816344 0 0) 0x7f3e5006b100 con 0x7f3e500a9920 2024-03-25T18:12:39.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.127+0000 7f3e4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:39.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.127+0000 7f3e4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:39.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.127+0000 7f3e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:39.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.127+0000 7f3e4effd640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:12:39.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.127+0000 7f3e4effd640 1 -- 172.21.15.81:0/1700190030 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e5013fec0 con 0x7f3e500a9920 2024-03-25T18:12:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e4effd640 1 -- 172.21.15.81:0/1700190030 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2426898887 0 0) 0x7f3e38003520 con 0x7f3e50146450 2024-03-25T18:12:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e4effd640 1 -- 172.21.15.81:0/1700190030 >> v1:172.21.15.63:6790/0 conn(0x7f3e500b45a0 legacy=0x7f3e50145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e4effd640 1 -- 172.21.15.81:0/1700190030 >> v1:172.21.15.63:6789/0 conn(0x7f3e500a9920 legacy=0x7f3e5013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e4effd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e4effd640 1 monclient: found mon.noname-c 2024-03-25T18:12:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e4effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:12:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e4effd640 1 -- 172.21.15.81:0/1700190030 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e50140050 con 0x7f3e50146450 2024-03-25T18:12:39.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e4effd640 10 monclient: _finish_auth 0 2024-03-25T18:12:39.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e4effd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:39.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:39.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e4effd640 1 -- 172.21.15.81:0/1700190030 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3e48001a10 con 0x7f3e50146450 2024-03-25T18:12:39.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e566d3640 5 monclient: authenticate success, global_id 4123 2024-03-25T18:12:39.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e566d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:39.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:39.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e4effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:39.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e4effd640 10 monclient: dump: 2024-03-25T18:12:39.131 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:39.131 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:39.131 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.131 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.131 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:39.132 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:39.132 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:39.132 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:39.132 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:39.132 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:39.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.129+0000 7f3e566d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:39.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.131+0000 7f3e4effd640 1 -- 172.21.15.81:0/1700190030 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e48001b70 con 0x7f3e50146450 2024-03-25T18:12:39.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.131+0000 7f3e4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:39.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.131+0000 7f3e566d3640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:39.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.131+0000 7f3e566d3640 4 set_mon_vals no callback set 2024-03-25T18:12:39.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.131+0000 7f3e4effd640 1 -- 172.21.15.81:0/1700190030 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3e48001de0 con 0x7f3e50146450 2024-03-25T18:12:39.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.131+0000 7f3e4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:39.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.131+0000 7f3e4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:39.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.131+0000 7f3e4effd640 10 monclient: dump: 2024-03-25T18:12:39.134 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:39.134 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:39.134 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.134 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.134 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:39.134 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:39.134 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:39.134 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:39.134 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:39.134 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:39.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.132+0000 7f3e566d3640 10 monclient: shutdown 2024-03-25T18:12:39.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.132+0000 7f3e566d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:39.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.132+0000 7f3e566d3640 1 -- 172.21.15.81:0/1700190030 >> v1:172.21.15.81:6789/0 conn(0x7f3e50146450 legacy=0x7f3e5014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:39.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.132+0000 7f3e566d3640 1 -- 172.21.15.81:0/1700190030 shutdown_connections 2024-03-25T18:12:39.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.132+0000 7f3e566d3640 1 -- 172.21.15.81:0/1700190030 >> 172.21.15.81:0/1700190030 conn(0x7f3e500b7f40 msgr2=0x7f3e500b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:39.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.132+0000 7f3e566d3640 1 -- 172.21.15.81:0/1700190030 shutdown_connections 2024-03-25T18:12:39.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.132+0000 7f3e566d3640 1 -- 172.21.15.81:0/1700190030 wait complete. 2024-03-25T18:12:39.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.133+0000 7f3e566d3640 10 monclient: build_initial_monmap 2024-03-25T18:12:39.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.133+0000 7f3e566d3640 10 monclient: monmap: 2024-03-25T18:12:39.135 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T18:12:39.135 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:39.135 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-25T18:12:39.135 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-25T18:12:39.135 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:39.135 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-25T18:12:39.135 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:39.135 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:12:39.135 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:12:39.135 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:39.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.133+0000 7f3e566d3640 1 librados: starting msgr at 2024-03-25T18:12:39.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.133+0000 7f3e566d3640 1 librados: starting objecter 2024-03-25T18:12:39.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.133+0000 7f3e566d3640 1 Processor -- start 2024-03-25T18:12:39.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.133+0000 7f3e566d3640 1 -- start start 2024-03-25T18:12:39.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.133+0000 7f3e566d3640 1 librados: setting wanted keys 2024-03-25T18:12:39.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.133+0000 7f3e566d3640 1 librados: calling monclient init 2024-03-25T18:12:39.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.133+0000 7f3e566d3640 10 monclient: init 2024-03-25T18:12:39.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.134+0000 7f3e566d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:39.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.134+0000 7f3e566d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:39.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.134+0000 7f3e566d3640 10 monclient(hunting): picked mon.noname-c con 0x7f3e500a9920 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:39.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.134+0000 7f3e566d3640 10 monclient(hunting): picked mon.noname-a con 0x7f3e500b45a0 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:39.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.134+0000 7f3e566d3640 10 monclient(hunting): picked mon.noname-b con 0x7f3e50146450 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:39.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.134+0000 7f3e566d3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e501442a0 con 0x7f3e500b45a0 2024-03-25T18:12:39.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.134+0000 7f3e566d3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e50143e30 con 0x7f3e50146450 2024-03-25T18:12:39.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.134+0000 7f3e566d3640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e50143fc0 con 0x7f3e500a9920 2024-03-25T18:12:39.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.134+0000 7f3e566d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:39.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.134+0000 7f3e566d3640 10 monclient(hunting): authenticate will time out at 7464.313859s 2024-03-25T18:12:39.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.134+0000 7f3e4ffff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f3e500a9920 0x7f3e501da870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.81:48216/0 (socket says 172.21.15.81:48216) 2024-03-25T18:12:39.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.134+0000 7f3e4ffff640 1 -- 172.21.15.81:0/3935175095 learned_addr learned my addr 172.21.15.81:0/3935175095 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T18:12:39.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.134+0000 7f3e4cff9640 10 client.?.objecter ms_handle_connect 0x7f3e500a9920 2024-03-25T18:12:39.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.134+0000 7f3e4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:39.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.135+0000 7f3e4cff9640 10 client.?.objecter ms_handle_connect 0x7f3e50146450 2024-03-25T18:12:39.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.135+0000 7f3e4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:39.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.135+0000 7f3e4cff9640 10 client.?.objecter ms_handle_connect 0x7f3e500b45a0 2024-03-25T18:12:39.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.135+0000 7f3e4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:39.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.136+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706022538 0 0) 0x7f3e50143e30 con 0x7f3e50146450 2024-03-25T18:12:39.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.136+0000 7f3e4cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:39.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.136+0000 7f3e4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:39.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.136+0000 7f3e4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:39.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.136+0000 7f3e4cff9640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:12:39.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.136+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e3c004eb0 con 0x7f3e50146450 2024-03-25T18:12:39.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.136+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908135323 0 0) 0x7f3e50143fc0 con 0x7f3e500a9920 2024-03-25T18:12:39.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.136+0000 7f3e4cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:39.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.136+0000 7f3e4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:39.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.136+0000 7f3e4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:39.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.136+0000 7f3e4cff9640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:12:39.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.136+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e50143e30 con 0x7f3e500a9920 2024-03-25T18:12:39.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.137+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953522928 0 0) 0x7f3e501442a0 con 0x7f3e500b45a0 2024-03-25T18:12:39.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.137+0000 7f3e4cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:39.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.137+0000 7f3e4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:39.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.137+0000 7f3e4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:39.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.137+0000 7f3e4cff9640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:12:39.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.137+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e50143fc0 con 0x7f3e500b45a0 2024-03-25T18:12:39.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.141+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1163185793 0 0) 0x7f3e3c004eb0 con 0x7f3e50146450 2024-03-25T18:12:39.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.141+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e501442a0 con 0x7f3e50146450 2024-03-25T18:12:39.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.141+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3e48001e80 con 0x7f3e50146450 2024-03-25T18:12:39.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.141+0000 7f3e4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:39.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.141+0000 7f3e4cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:39.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.141+0000 7f3e4cff9640 10 monclient(hunting): dump: 2024-03-25T18:12:39.143 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:39.143 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:39.143 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.143 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.143 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:39.143 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:39.143 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:39.143 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:39.143 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:39.143 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:39.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.141+0000 7f3e4cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:39.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.141+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2172423084 0 0) 0x7f3e50143e30 con 0x7f3e500a9920 2024-03-25T18:12:39.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.141+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e3c004eb0 con 0x7f3e500a9920 2024-03-25T18:12:39.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.141+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3e440019f0 con 0x7f3e500a9920 2024-03-25T18:12:39.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.141+0000 7f3e4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:39.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.141+0000 7f3e4cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:39.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.141+0000 7f3e4cff9640 10 monclient(hunting): dump: 2024-03-25T18:12:39.144 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:39.144 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:39.144 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.144 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.144 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:39.144 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:39.144 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:39.144 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:39.144 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:39.144 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:39.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.142+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4113620343 0 0) 0x7f3e50143fc0 con 0x7f3e500b45a0 2024-03-25T18:12:39.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.142+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e50143e30 con 0x7f3e500b45a0 2024-03-25T18:12:39.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.142+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3e40001930 con 0x7f3e500b45a0 2024-03-25T18:12:39.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.142+0000 7f3e4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:39.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.142+0000 7f3e4cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:39.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.142+0000 7f3e4cff9640 10 monclient(hunting): dump: 2024-03-25T18:12:39.146 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:39.146 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:39.147 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.147 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.147 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:39.147 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:39.147 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:39.147 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:39.147 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:39.147 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:39.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2985616041 0 0) 0x7f3e3c004eb0 con 0x7f3e500a9920 2024-03-25T18:12:39.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 >> v1:172.21.15.81:6789/0 conn(0x7f3e50146450 legacy=0x7f3e501e1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:39.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 >> v1:172.21.15.63:6789/0 conn(0x7f3e500b45a0 legacy=0x7f3e501daf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:39.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e4cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:12:39.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e4cff9640 1 monclient: found mon.c 2024-03-25T18:12:39.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e4cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:39.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e4cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:39.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e50145210 con 0x7f3e500a9920 2024-03-25T18:12:39.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e4cff9640 10 monclient: _finish_auth 0 2024-03-25T18:12:39.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e4cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:12:39.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:39.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e566d3640 5 monclient: authenticate success, global_id 4130 2024-03-25T18:12:39.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e566d3640 10 monclient: _renew_subs 2024-03-25T18:12:39.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e566d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:39.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e566d3640 1 -- 172.21.15.81:0/3935175095 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e50145460 con 0x7f3e500a9920 2024-03-25T18:12:39.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e566d3640 10 client.4130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:39.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e566d3640 10 monclient: _renew_subs 2024-03-25T18:12:39.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e566d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:39.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e566d3640 1 -- 172.21.15.81:0/3935175095 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e50144e20 con 0x7f3e500a9920 2024-03-25T18:12:39.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.144+0000 7f3e566d3640 1 librados: init done 2024-03-25T18:12:39.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.145+0000 7f3e566d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:39.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.145+0000 7f3e566d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:39.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.145+0000 7f3e566d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:39.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.145+0000 7f3e566d3640 1 -- 172.21.15.81:0/3935175095 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e5004f730 con 0x7f3e500a9920 2024-03-25T18:12:39.159 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:39.159+0000 7f0c0da1f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:12:39.217 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:39.217+0000 7fd9c8010040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:12:39.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.248+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e44003610 con 0x7f3e500a9920 2024-03-25T18:12:39.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.248+0000 7f3e4cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:39.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.248+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3e440039a0 con 0x7f3e500a9920 2024-03-25T18:12:39.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.248+0000 7f3e4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:39.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.248+0000 7f3e4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:39.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.248+0000 7f3e4cff9640 10 monclient: dump: 2024-03-25T18:12:39.249 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:39.249 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:39.250 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.250 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.250 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:39.250 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:39.250 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:39.250 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:39.250 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:39.250 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:39.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.248+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3e44003f50 con 0x7f3e500a9920 2024-03-25T18:12:39.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.248+0000 7f3e4e7fc640 4 set_mon_vals no callback set 2024-03-25T18:12:39.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.249+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2384695710 0 0) 0x7f3e440065f0 con 0x7f3e500a9920 2024-03-25T18:12:39.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.249+0000 7f3e4cff9640 10 client.4130.objecter ms_dispatch 0x7f3e50000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:12:39.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.249+0000 7f3e4cff9640 3 client.4130.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:12:39.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.249+0000 7f3e4cff9640 3 client.4130.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:12:39.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.249+0000 7f3e4cff9640 20 client.4130.objecter dump_active .. 0 homeless 2024-03-25T18:12:39.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.249+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3e44004470 con 0x7f3e500a9920 2024-03-25T18:12:39.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.249+0000 7f3e4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:39.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.249+0000 7f3e4cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:39.253 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:39.252+0000 7f0c0da1f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:12:39.266 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:39.265+0000 7fd9c8010040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:12:39.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.322+0000 7f3e566d3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cdfc79b2-cb55-4ddb-bfac-682073ac86dc", "id": 2}] 2024-03-25T18:12:39.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.322+0000 7f3e566d3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cdfc79b2-cb55-4ddb-bfac-682073ac86dc", "id": 2}] 2024-03-25T18:12:39.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.322+0000 7f3e566d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:39.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.322+0000 7f3e566d3640 1 -- 172.21.15.81:0/3935175095 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd new", "uuid": "cdfc79b2-cb55-4ddb-bfac-682073ac86dc", "id": 2} v 0) v1 -- 0x7f3e5013b4d0 con 0x7f3e500a9920 2024-03-25T18:12:39.388 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:39.387+0000 7fd9c8010040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:12:39.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.389+0000 7f3e4cff9640 1 -- 172.21.15.81:0/3935175095 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cdfc79b2-cb55-4ddb-bfac-682073ac86dc", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1880102519 0 216599125) 0x7f3e44006320 con 0x7f3e500a9920 2024-03-25T18:12:39.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.389+0000 7f3e4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cdfc79b2-cb55-4ddb-bfac-682073ac86dc", "id": 2}] 2024-03-25T18:12:39.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.389+0000 7f3e4cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:39.390 INFO:teuthology.orchestra.run.smithi081.stdout:2 2024-03-25T18:12:39.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.389+0000 7f3e566d3640 10 librados: watch_flush enter 2024-03-25T18:12:39.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.390+0000 7f3e566d3640 10 librados: watch_flush exit 2024-03-25T18:12:39.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.390+0000 7f3e566d3640 20 client.4130.objecter shutdown clearing up homeless session... 2024-03-25T18:12:39.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.390+0000 7f3e566d3640 10 client.4130.objecter successfully canceled tick 2024-03-25T18:12:39.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.390+0000 7f3e566d3640 10 monclient: shutdown 2024-03-25T18:12:39.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.390+0000 7f3e566d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:39.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.390+0000 7f3e566d3640 1 -- 172.21.15.81:0/3935175095 >> v1:172.21.15.63:6790/0 conn(0x7f3e500a9920 legacy=0x7f3e501da870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:39.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.390+0000 7f3e566d3640 1 -- 172.21.15.81:0/3935175095 shutdown_connections 2024-03-25T18:12:39.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.390+0000 7f3e566d3640 1 -- 172.21.15.81:0/3935175095 >> 172.21.15.81:0/3935175095 conn(0x7f3e500b7f40 msgr2=0x7f3e50074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:39.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.390+0000 7f3e566d3640 1 -- 172.21.15.81:0/3935175095 shutdown_connections 2024-03-25T18:12:39.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.390+0000 7f3e566d3640 1 -- 172.21.15.81:0/3935175095 wait complete. 2024-03-25T18:12:39.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.390+0000 7f3e566d3640 1 librados: shutdown 2024-03-25T18:12:39.401 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new d53bda7b-3db2-454a-a647-50c86732edcf 3 2024-03-25T18:12:39.412 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:39.411+0000 7f0c0da1f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:12:39.457 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:39.456+0000 7f0c0da1f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:12:39.490 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:39.489+0000 7fd9c8010040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:12:39.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.512+0000 7f56d9a3e640 10 monclient: get_monmap_and_config 2024-03-25T18:12:39.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.512+0000 7f56d9a3e640 10 monclient: build_initial_monmap 2024-03-25T18:12:39.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.512+0000 7f56d9a3e640 10 monclient: monmap: 2024-03-25T18:12:39.513 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T18:12:39.513 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:39.513 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:12:39.513531+0000 2024-03-25T18:12:39.513 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:12:39.513531+0000 2024-03-25T18:12:39.513 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:39.513 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:39.513 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:39.513 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:12:39.514 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:12:39.514 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:39.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.513+0000 7f56d9a3e640 1 Processor -- start 2024-03-25T18:12:39.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.513+0000 7f56d9a3e640 1 -- start start 2024-03-25T18:12:39.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.513+0000 7f56d9a3e640 10 monclient: init 2024-03-25T18:12:39.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.513+0000 7f56d9a3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:39.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.513+0000 7f56d9a3e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:39.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.513+0000 7f56d9a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f56d40a9920 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:39.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.513+0000 7f56d9a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f56d40b45a0 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:39.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.513+0000 7f56d9a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f56d4146450 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:39.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.513+0000 7f56d9a3e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d406b100 con 0x7f56d40b45a0 2024-03-25T18:12:39.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.513+0000 7f56d9a3e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d40b2940 con 0x7f56d40a9920 2024-03-25T18:12:39.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.513+0000 7f56d9a3e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d413fec0 con 0x7f56d4146450 2024-03-25T18:12:39.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.513+0000 7f56d9a3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:39.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.513+0000 7f56d9a3e640 10 monclient(hunting): authenticate will time out at 7464.693369s 2024-03-25T18:12:39.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.514+0000 7f56d37fe640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f56d4146450 0x7f56d414dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.81:48220/0 (socket says 172.21.15.81:48220) 2024-03-25T18:12:39.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.514+0000 7f56d37fe640 1 -- 172.21.15.81:0/2568447170 learned_addr learned my addr 172.21.15.81:0/2568447170 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T18:12:39.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 1 -- 172.21.15.81:0/2568447170 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509624159 0 0) 0x7f56d406b100 con 0x7f56d40b45a0 2024-03-25T18:12:39.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:39.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:39.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:39.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:12:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 1 -- 172.21.15.81:0/2568447170 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56bc003520 con 0x7f56d40b45a0 2024-03-25T18:12:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 1 -- 172.21.15.81:0/2568447170 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326547135 0 0) 0x7f56d40b2940 con 0x7f56d40a9920 2024-03-25T18:12:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:12:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 1 -- 172.21.15.81:0/2568447170 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d406b100 con 0x7f56d40a9920 2024-03-25T18:12:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 1 -- 172.21.15.81:0/2568447170 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587360960 0 0) 0x7f56d413fec0 con 0x7f56d4146450 2024-03-25T18:12:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:12:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.516+0000 7f56d1ffb640 1 -- 172.21.15.81:0/2568447170 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d40b2940 con 0x7f56d4146450 2024-03-25T18:12:39.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d1ffb640 1 -- 172.21.15.81:0/2568447170 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1325696687 0 0) 0x7f56d406b100 con 0x7f56d40a9920 2024-03-25T18:12:39.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d1ffb640 1 -- 172.21.15.81:0/2568447170 >> v1:172.21.15.63:6790/0 conn(0x7f56d4146450 legacy=0x7f56d414dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:39.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d1ffb640 1 -- 172.21.15.81:0/2568447170 >> v1:172.21.15.63:6789/0 conn(0x7f56d40b45a0 legacy=0x7f56d4145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:39.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:12:39.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d1ffb640 1 monclient: found mon.noname-c 2024-03-25T18:12:39.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:39.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:12:39.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d1ffb640 1 -- 172.21.15.81:0/2568447170 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d4140050 con 0x7f56d40a9920 2024-03-25T18:12:39.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:12:39.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:12:39.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:39.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d1ffb640 1 -- 172.21.15.81:0/2568447170 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f56c4001670 con 0x7f56d40a9920 2024-03-25T18:12:39.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:39.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:39.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d1ffb640 10 monclient: dump: 2024-03-25T18:12:39.521 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:39.521 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:39.521 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.521 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.521 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:39.521 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:39.521 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:39.521 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:39.522 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:39.522 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:39.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d9a3e640 5 monclient: authenticate success, global_id 4129 2024-03-25T18:12:39.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.519+0000 7f56d9a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:39.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.521+0000 7f56d1ffb640 1 -- 172.21.15.81:0/2568447170 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56c40017d0 con 0x7f56d40a9920 2024-03-25T18:12:39.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.521+0000 7f56d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:39.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.521+0000 7f56d1ffb640 1 -- 172.21.15.81:0/2568447170 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f56c4001a40 con 0x7f56d40a9920 2024-03-25T18:12:39.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.521+0000 7f56d9a3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:39.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.521+0000 7f56d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:39.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.521+0000 7f56d9a3e640 4 set_mon_vals no callback set 2024-03-25T18:12:39.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.521+0000 7f56d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:39.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.521+0000 7f56d1ffb640 10 monclient: dump: 2024-03-25T18:12:39.524 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:39.524 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:39.524 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.524 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.524 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:39.524 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:39.524 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:39.524 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:39.524 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:39.524 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:39.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.521+0000 7f56d9a3e640 10 monclient: shutdown 2024-03-25T18:12:39.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.521+0000 7f56d9a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.521+0000 7f56d9a3e640 1 -- 172.21.15.81:0/2568447170 >> v1:172.21.15.81:6789/0 conn(0x7f56d40a9920 legacy=0x7f56d413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.521+0000 7f56d9a3e640 1 -- 172.21.15.81:0/2568447170 shutdown_connections 2024-03-25T18:12:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.521+0000 7f56d9a3e640 1 -- 172.21.15.81:0/2568447170 >> 172.21.15.81:0/2568447170 conn(0x7f56d40b7f40 msgr2=0x7f56d40b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.521+0000 7f56d9a3e640 1 -- 172.21.15.81:0/2568447170 shutdown_connections 2024-03-25T18:12:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.521+0000 7f56d9a3e640 1 -- 172.21.15.81:0/2568447170 wait complete. 2024-03-25T18:12:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.522+0000 7f56d9a3e640 10 monclient: build_initial_monmap 2024-03-25T18:12:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.522+0000 7f56d9a3e640 10 monclient: monmap: 2024-03-25T18:12:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T18:12:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-25T18:12:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-25T18:12:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-25T18:12:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:12:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:12:39.526 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:39.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.522+0000 7f56d9a3e640 1 librados: starting msgr at 2024-03-25T18:12:39.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.522+0000 7f56d9a3e640 1 librados: starting objecter 2024-03-25T18:12:39.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.522+0000 7f56d9a3e640 1 Processor -- start 2024-03-25T18:12:39.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.523+0000 7f56d9a3e640 1 -- start start 2024-03-25T18:12:39.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.523+0000 7f56d9a3e640 1 librados: setting wanted keys 2024-03-25T18:12:39.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.523+0000 7f56d9a3e640 1 librados: calling monclient init 2024-03-25T18:12:39.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.523+0000 7f56d9a3e640 10 monclient: init 2024-03-25T18:12:39.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.523+0000 7f56d9a3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:39.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.523+0000 7f56d9a3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:39.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.523+0000 7f56d9a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f56d40a9920 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:39.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.523+0000 7f56d9a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f56d40b45a0 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:39.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.523+0000 7f56d9a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f56d4146450 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:39.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.523+0000 7f56d9a3e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d40abe20 con 0x7f56d40b45a0 2024-03-25T18:12:39.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.523+0000 7f56d9a3e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d41d6f70 con 0x7f56d4146450 2024-03-25T18:12:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.523+0000 7f56d9a3e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d4144890 con 0x7f56d40a9920 2024-03-25T18:12:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.523+0000 7f56d9a3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.523+0000 7f56d9a3e640 10 monclient(hunting): authenticate will time out at 7464.703352s 2024-03-25T18:12:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.524+0000 7f56d2ffd640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f56d40a9920 0x7f56d4144180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.81:48228/0 (socket says 172.21.15.81:48228) 2024-03-25T18:12:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.524+0000 7f56d2ffd640 1 -- 172.21.15.81:0/1970446467 learned_addr learned my addr 172.21.15.81:0/1970446467 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T18:12:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.524+0000 7f56bbfff640 10 client.?.objecter ms_handle_connect 0x7f56d40a9920 2024-03-25T18:12:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.524+0000 7f56bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.524+0000 7f56bbfff640 10 client.?.objecter ms_handle_connect 0x7f56d4146450 2024-03-25T18:12:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.524+0000 7f56bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.526+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489950725 0 0) 0x7f56d41d6f70 con 0x7f56d4146450 2024-03-25T18:12:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.526+0000 7f56bbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.526+0000 7f56bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.526+0000 7f56bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.526+0000 7f56bbfff640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:12:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.526+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56c0004eb0 con 0x7f56d4146450 2024-03-25T18:12:39.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.526+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126297762 0 0) 0x7f56d4144890 con 0x7f56d40a9920 2024-03-25T18:12:39.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.526+0000 7f56bbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:39.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.526+0000 7f56bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:39.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.526+0000 7f56bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:39.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.526+0000 7f56bbfff640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:12:39.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.526+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d41d6f70 con 0x7f56d40a9920 2024-03-25T18:12:39.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.528+0000 7f56bbfff640 10 client.?.objecter ms_handle_connect 0x7f56d40b45a0 2024-03-25T18:12:39.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.528+0000 7f56bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:39.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.530+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159882943 0 0) 0x7f56d40abe20 con 0x7f56d40b45a0 2024-03-25T18:12:39.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.530+0000 7f56bbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:39.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.530+0000 7f56bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:39.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.530+0000 7f56bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:39.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.530+0000 7f56bbfff640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:12:39.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.530+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d4144890 con 0x7f56d40b45a0 2024-03-25T18:12:39.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.530+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 286401301 0 0) 0x7f56c0004eb0 con 0x7f56d4146450 2024-03-25T18:12:39.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.530+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56d40abe20 con 0x7f56d4146450 2024-03-25T18:12:39.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.530+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f56cc001bd0 con 0x7f56d4146450 2024-03-25T18:12:39.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.530+0000 7f56bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:39.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.530+0000 7f56bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:39.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.530+0000 7f56bbfff640 10 monclient(hunting): dump: 2024-03-25T18:12:39.532 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:39.532 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:39.532 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.532 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.532 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:39.532 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:39.532 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:39.532 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:39.533 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:39.533 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:39.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.530+0000 7f56bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:39.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.532+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 37700285 0 0) 0x7f56d41d6f70 con 0x7f56d40a9920 2024-03-25T18:12:39.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.532+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56c0004eb0 con 0x7f56d40a9920 2024-03-25T18:12:39.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.532+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f56c4001ae0 con 0x7f56d40a9920 2024-03-25T18:12:39.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.532+0000 7f56bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:39.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.532+0000 7f56bbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:39.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.532+0000 7f56bbfff640 10 monclient(hunting): dump: 2024-03-25T18:12:39.534 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:39.534 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:39.534 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.534 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.534 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:39.534 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:39.534 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:39.534 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:39.534 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:39.534 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:39.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 740788725 0 0) 0x7f56d40abe20 con 0x7f56d4146450 2024-03-25T18:12:39.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 >> v1:172.21.15.63:6790/0 conn(0x7f56d40a9920 legacy=0x7f56d4144180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:39.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 >> v1:172.21.15.63:6789/0 conn(0x7f56d40b45a0 legacy=0x7f56d41d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:39.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56bbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:39.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56bbfff640 1 monclient: found mon.b 2024-03-25T18:12:39.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:39.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56bbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:39.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d4144a20 con 0x7f56d4146450 2024-03-25T18:12:39.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56bbfff640 10 monclient: _finish_auth 0 2024-03-25T18:12:39.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56bbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:39.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:39.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56d9a3e640 5 monclient: authenticate success, global_id 4132 2024-03-25T18:12:39.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56d9a3e640 10 monclient: _renew_subs 2024-03-25T18:12:39.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56d9a3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:39.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56d9a3e640 1 -- 172.21.15.81:0/1970446467 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d4140ac0 con 0x7f56d4146450 2024-03-25T18:12:39.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56d9a3e640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:39.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56d9a3e640 10 monclient: _renew_subs 2024-03-25T18:12:39.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56d9a3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:39.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56d9a3e640 1 -- 172.21.15.81:0/1970446467 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d41435b0 con 0x7f56d4146450 2024-03-25T18:12:39.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.534+0000 7f56d9a3e640 1 librados: init done 2024-03-25T18:12:39.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.535+0000 7f56d9a3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:39.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.535+0000 7f56d9a3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:39.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.535+0000 7f56d9a3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:39.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.535+0000 7f56d9a3e640 1 -- 172.21.15.81:0/1970446467 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d40abc40 con 0x7f56d4146450 2024-03-25T18:12:39.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.557+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56cc001fe0 con 0x7f56d4146450 2024-03-25T18:12:39.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.557+0000 7f56bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:39.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.557+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f56cc004d00 con 0x7f56d4146450 2024-03-25T18:12:39.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.557+0000 7f56bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:39.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.557+0000 7f56bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:39.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.557+0000 7f56bbfff640 10 monclient: dump: 2024-03-25T18:12:39.559 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:39.559 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:39.559 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.559 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:39.559 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:39.559 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:39.559 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:39.559 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:39.559 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:39.559 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:39.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.557+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f56cc0052b0 con 0x7f56d4146450 2024-03-25T18:12:39.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.557+0000 7f56d17fa640 4 set_mon_vals no callback set 2024-03-25T18:12:39.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.557+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 4099116253 0 0) 0x7f56cc0067c0 con 0x7f56d4146450 2024-03-25T18:12:39.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.557+0000 7f56bbfff640 10 client.4132.objecter ms_dispatch 0x7f56d4000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:12:39.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.557+0000 7f56bbfff640 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:12:39.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.557+0000 7f56bbfff640 3 client.4132.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:12:39.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.557+0000 7f56bbfff640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-25T18:12:39.617 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:39.617+0000 7f0c0da1f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:12:39.657 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:39.656+0000 7fd9c8010040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:12:39.701 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:39.700+0000 7fd9c8010040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:12:39.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.730+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f56cc006460 con 0x7f56d4146450 2024-03-25T18:12:39.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.730+0000 7f56bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:39.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.730+0000 7f56bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.824+0000 7f56d9a3e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d53bda7b-3db2-454a-a647-50c86732edcf", "id": 3}] 2024-03-25T18:12:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.824+0000 7f56d9a3e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d53bda7b-3db2-454a-a647-50c86732edcf", "id": 3}] 2024-03-25T18:12:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.824+0000 7f56d9a3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.824+0000 7f56d9a3e640 1 -- 172.21.15.81:0/1970446467 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d53bda7b-3db2-454a-a647-50c86732edcf", "id": 3} v 0) v1 -- 0x7f56d413b4d0 con 0x7f56d4146450 2024-03-25T18:12:39.867 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:39.867+0000 7fd9c8010040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:12:39.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.943+0000 7f56bbfff640 1 -- 172.21.15.81:0/1970446467 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d53bda7b-3db2-454a-a647-50c86732edcf", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 913829274 0 525046306) 0x7f56cc005bf0 con 0x7f56d4146450 2024-03-25T18:12:39.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.943+0000 7f56bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d53bda7b-3db2-454a-a647-50c86732edcf", "id": 3}] 2024-03-25T18:12:39.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.943+0000 7f56bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:39.944 INFO:teuthology.orchestra.run.smithi081.stdout:3 2024-03-25T18:12:39.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.944+0000 7f56d9a3e640 10 librados: watch_flush enter 2024-03-25T18:12:39.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.944+0000 7f56d9a3e640 10 librados: watch_flush exit 2024-03-25T18:12:39.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.944+0000 7f56d9a3e640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-25T18:12:39.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.944+0000 7f56d9a3e640 10 client.4132.objecter successfully canceled tick 2024-03-25T18:12:39.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.944+0000 7f56d9a3e640 10 monclient: shutdown 2024-03-25T18:12:39.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.944+0000 7f56d9a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:39.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.944+0000 7f56d9a3e640 1 -- 172.21.15.81:0/1970446467 >> v1:172.21.15.81:6789/0 conn(0x7f56d4146450 legacy=0x7f56d41370e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:39.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.944+0000 7f56d9a3e640 1 -- 172.21.15.81:0/1970446467 shutdown_connections 2024-03-25T18:12:39.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.944+0000 7f56d9a3e640 1 -- 172.21.15.81:0/1970446467 >> 172.21.15.81:0/1970446467 conn(0x7f56d40b7f40 msgr2=0x7f56d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:39.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.944+0000 7f56d9a3e640 1 -- 172.21.15.81:0/1970446467 shutdown_connections 2024-03-25T18:12:39.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.944+0000 7f56d9a3e640 1 -- 172.21.15.81:0/1970446467 wait complete. 2024-03-25T18:12:39.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:39.944+0000 7f56d9a3e640 1 librados: shutdown 2024-03-25T18:12:39.955 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new b0b711e9-7872-4700-83a6-986664c998d5 4 2024-03-25T18:12:40.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.053+0000 7fbb778dd640 10 monclient: get_monmap_and_config 2024-03-25T18:12:40.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.054+0000 7fbb778dd640 10 monclient: build_initial_monmap 2024-03-25T18:12:40.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.054+0000 7fbb778dd640 10 monclient: monmap: 2024-03-25T18:12:40.055 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T18:12:40.055 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.055 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:12:40.054996+0000 2024-03-25T18:12:40.055 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:12:40.054996+0000 2024-03-25T18:12:40.055 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:40.055 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.055 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:40.055 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:12:40.055 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:12:40.055 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.054+0000 7fbb778dd640 1 Processor -- start 2024-03-25T18:12:40.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.054+0000 7fbb778dd640 1 -- start start 2024-03-25T18:12:40.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.054+0000 7fbb778dd640 10 monclient: init 2024-03-25T18:12:40.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.055+0000 7fbb778dd640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:40.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.055+0000 7fbb778dd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:40.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.055+0000 7fbb778dd640 10 monclient(hunting): picked mon.noname-a con 0x7fbb70066b50 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:40.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.055+0000 7fbb778dd640 10 monclient(hunting): picked mon.noname-c con 0x7fbb70142290 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:40.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.055+0000 7fbb778dd640 10 monclient(hunting): picked mon.noname-b con 0x7fbb7014bba0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:40.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.055+0000 7fbb778dd640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb7006fb90 con 0x7fbb70066b50 2024-03-25T18:12:40.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.055+0000 7fbb778dd640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb70146190 con 0x7fbb70142290 2024-03-25T18:12:40.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.055+0000 7fbb778dd640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb70146620 con 0x7fbb7014bba0 2024-03-25T18:12:40.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.055+0000 7fbb778dd640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:40.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.055+0000 7fbb778dd640 10 monclient(hunting): authenticate will time out at 7465.235070s 2024-03-25T18:12:40.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.055+0000 7fbb75e7a640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fbb7014bba0 0x7fbb7014df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.81:48240/0 (socket says 172.21.15.81:48240) 2024-03-25T18:12:40.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.055+0000 7fbb75e7a640 1 -- 172.21.15.81:0/2392257408 learned_addr learned my addr 172.21.15.81:0/2392257408 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T18:12:40.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.058+0000 7fbb67fff640 1 -- 172.21.15.81:0/2392257408 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43482871 0 0) 0x7fbb70146190 con 0x7fbb70142290 2024-03-25T18:12:40.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.058+0000 7fbb67fff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.058+0000 7fbb67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.058+0000 7fbb67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.058+0000 7fbb67fff640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:12:40.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.058+0000 7fbb67fff640 1 -- 172.21.15.81:0/2392257408 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb5c003520 con 0x7fbb70142290 2024-03-25T18:12:40.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.058+0000 7fbb67fff640 1 -- 172.21.15.81:0/2392257408 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340401710 0 0) 0x7fbb7006fb90 con 0x7fbb70066b50 2024-03-25T18:12:40.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.058+0000 7fbb67fff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.058+0000 7fbb67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.058+0000 7fbb67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.058+0000 7fbb67fff640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:12:40.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.058+0000 7fbb67fff640 1 -- 172.21.15.81:0/2392257408 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb70146190 con 0x7fbb70066b50 2024-03-25T18:12:40.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.059+0000 7fbb67fff640 1 -- 172.21.15.81:0/2392257408 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381441253 0 0) 0x7fbb70146620 con 0x7fbb7014bba0 2024-03-25T18:12:40.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.059+0000 7fbb67fff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.059+0000 7fbb67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.059+0000 7fbb67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.059+0000 7fbb67fff640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:12:40.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.059+0000 7fbb67fff640 1 -- 172.21.15.81:0/2392257408 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb7006fb90 con 0x7fbb7014bba0 2024-03-25T18:12:40.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb67fff640 1 -- 172.21.15.81:0/2392257408 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1316091889 0 0) 0x7fbb70146190 con 0x7fbb70066b50 2024-03-25T18:12:40.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb67fff640 1 -- 172.21.15.81:0/2392257408 >> v1:172.21.15.63:6790/0 conn(0x7fbb7014bba0 legacy=0x7fbb7014df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb67fff640 1 -- 172.21.15.81:0/2392257408 >> v1:172.21.15.81:6789/0 conn(0x7fbb70142290 legacy=0x7fbb70142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb67fff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb67fff640 1 monclient: found mon.noname-a 2024-03-25T18:12:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb67fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:12:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb67fff640 1 -- 172.21.15.81:0/2392257408 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7013fdd0 con 0x7fbb70066b50 2024-03-25T18:12:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb67fff640 10 monclient: _finish_auth 0 2024-03-25T18:12:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb67fff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb67fff640 1 -- 172.21.15.81:0/2392257408 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fbb6c001670 con 0x7fbb70066b50 2024-03-25T18:12:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb778dd640 5 monclient: authenticate success, global_id 4137 2024-03-25T18:12:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb778dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb67fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb67fff640 10 monclient: dump: 2024-03-25T18:12:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.064 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.061+0000 7fbb778dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:40.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.063+0000 7fbb67fff640 1 -- 172.21.15.81:0/2392257408 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb6c0017d0 con 0x7fbb70066b50 2024-03-25T18:12:40.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.063+0000 7fbb67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:40.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.063+0000 7fbb67fff640 1 -- 172.21.15.81:0/2392257408 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fbb6c001ba0 con 0x7fbb70066b50 2024-03-25T18:12:40.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.063+0000 7fbb67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.063+0000 7fbb67fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:40.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.063+0000 7fbb67fff640 10 monclient: dump: 2024-03-25T18:12:40.066 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.066 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.066 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.066 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.067 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.063+0000 7fbb778dd640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.063+0000 7fbb778dd640 4 set_mon_vals no callback set 2024-03-25T18:12:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.064+0000 7fbb778dd640 10 monclient: shutdown 2024-03-25T18:12:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.064+0000 7fbb778dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.064+0000 7fbb778dd640 1 -- 172.21.15.81:0/2392257408 >> v1:172.21.15.63:6789/0 conn(0x7fbb70066b50 legacy=0x7fbb7013f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.064+0000 7fbb778dd640 1 -- 172.21.15.81:0/2392257408 shutdown_connections 2024-03-25T18:12:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.064+0000 7fbb778dd640 1 -- 172.21.15.81:0/2392257408 >> 172.21.15.81:0/2392257408 conn(0x7fbb700b46e0 msgr2=0x7fbb700b4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.064+0000 7fbb778dd640 1 -- 172.21.15.81:0/2392257408 shutdown_connections 2024-03-25T18:12:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.064+0000 7fbb778dd640 1 -- 172.21.15.81:0/2392257408 wait complete. 2024-03-25T18:12:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.065+0000 7fbb778dd640 10 monclient: build_initial_monmap 2024-03-25T18:12:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.065+0000 7fbb778dd640 10 monclient: monmap: 2024-03-25T18:12:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T18:12:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-25T18:12:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-25T18:12:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-25T18:12:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:12:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:12:40.068 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.065+0000 7fbb778dd640 1 librados: starting msgr at 2024-03-25T18:12:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.065+0000 7fbb778dd640 1 librados: starting objecter 2024-03-25T18:12:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.065+0000 7fbb778dd640 1 Processor -- start 2024-03-25T18:12:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.065+0000 7fbb778dd640 1 -- start start 2024-03-25T18:12:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.065+0000 7fbb778dd640 1 librados: setting wanted keys 2024-03-25T18:12:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.065+0000 7fbb778dd640 1 librados: calling monclient init 2024-03-25T18:12:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.065+0000 7fbb778dd640 10 monclient: init 2024-03-25T18:12:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.066+0000 7fbb778dd640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.066+0000 7fbb778dd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.066+0000 7fbb778dd640 10 monclient(hunting): picked mon.noname-b con 0x7fbb70066b50 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.066+0000 7fbb778dd640 10 monclient(hunting): picked mon.noname-a con 0x7fbb70142290 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.066+0000 7fbb778dd640 10 monclient(hunting): picked mon.noname-c con 0x7fbb7014bba0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.066+0000 7fbb778dd640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb7006eaf0 con 0x7fbb70142290 2024-03-25T18:12:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.066+0000 7fbb778dd640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb70143d20 con 0x7fbb70066b50 2024-03-25T18:12:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.066+0000 7fbb778dd640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb70143f70 con 0x7fbb7014bba0 2024-03-25T18:12:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.066+0000 7fbb778dd640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.066+0000 7fbb778dd640 10 monclient(hunting): authenticate will time out at 7465.246014s 2024-03-25T18:12:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.066+0000 7fbb75e7a640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fbb7014bba0 0x7fbb701d7540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.81:48244/0 (socket says 172.21.15.81:48244) 2024-03-25T18:12:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.066+0000 7fbb75e7a640 1 -- 172.21.15.81:0/2649214247 learned_addr learned my addr 172.21.15.81:0/2649214247 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T18:12:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.067+0000 7fbb65ffb640 10 client.?.objecter ms_handle_connect 0x7fbb70066b50 2024-03-25T18:12:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.067+0000 7fbb65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.067+0000 7fbb65ffb640 10 client.?.objecter ms_handle_connect 0x7fbb7014bba0 2024-03-25T18:12:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.067+0000 7fbb65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.067+0000 7fbb65ffb640 10 client.?.objecter ms_handle_connect 0x7fbb70142290 2024-03-25T18:12:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.067+0000 7fbb65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197350113 0 0) 0x7fbb70143d20 con 0x7fbb70066b50 2024-03-25T18:12:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:12:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb54004eb0 con 0x7fbb70066b50 2024-03-25T18:12:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940845456 0 0) 0x7fbb7006eaf0 con 0x7fbb70142290 2024-03-25T18:12:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:12:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb70143d20 con 0x7fbb70142290 2024-03-25T18:12:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834548100 0 0) 0x7fbb70143f70 con 0x7fbb7014bba0 2024-03-25T18:12:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:12:40.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.069+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb7006eaf0 con 0x7fbb7014bba0 2024-03-25T18:12:40.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.073+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 22544576 0 0) 0x7fbb54004eb0 con 0x7fbb70066b50 2024-03-25T18:12:40.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.073+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb70143f70 con 0x7fbb70066b50 2024-03-25T18:12:40.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.073+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fbb6c001ae0 con 0x7fbb70066b50 2024-03-25T18:12:40.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.073+0000 7fbb65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.073+0000 7fbb65ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:40.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.073+0000 7fbb65ffb640 10 monclient(hunting): dump: 2024-03-25T18:12:40.075 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.075 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.075 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.075 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.075 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.075 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.075 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.075 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.075 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.075 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.073+0000 7fbb65ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:40.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.074+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1445886144 0 0) 0x7fbb70143d20 con 0x7fbb70142290 2024-03-25T18:12:40.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.074+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb54004eb0 con 0x7fbb70142290 2024-03-25T18:12:40.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.074+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fbb60001bc0 con 0x7fbb70142290 2024-03-25T18:12:40.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.074+0000 7fbb65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.074+0000 7fbb65ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:40.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.074+0000 7fbb65ffb640 10 monclient(hunting): dump: 2024-03-25T18:12:40.076 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.076 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.076 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.076 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.076 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.076 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.076 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.076 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.076 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.076 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.076+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4191696551 0 0) 0x7fbb7006eaf0 con 0x7fbb7014bba0 2024-03-25T18:12:40.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.076+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb70143d20 con 0x7fbb7014bba0 2024-03-25T18:12:40.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.076+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fbb68001b90 con 0x7fbb7014bba0 2024-03-25T18:12:40.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.076+0000 7fbb65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.076+0000 7fbb65ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:40.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.076+0000 7fbb65ffb640 10 monclient(hunting): dump: 2024-03-25T18:12:40.078 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.078 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.078 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.078 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.078 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.079 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.079 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.079 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.079 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.079 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3338962434 0 0) 0x7fbb70143f70 con 0x7fbb70066b50 2024-03-25T18:12:40.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 >> v1:172.21.15.63:6790/0 conn(0x7fbb7014bba0 legacy=0x7fbb701d7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 >> v1:172.21.15.63:6789/0 conn(0x7fbb70142290 legacy=0x7fbb70136760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb65ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:12:40.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb65ffb640 1 monclient: found mon.b 2024-03-25T18:12:40.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:40.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb65ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:40.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb70145460 con 0x7fbb70066b50 2024-03-25T18:12:40.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb65ffb640 10 monclient: _finish_auth 0 2024-03-25T18:12:40.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb65ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:12:40.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:40.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb778dd640 5 monclient: authenticate success, global_id 4138 2024-03-25T18:12:40.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb778dd640 10 monclient: _renew_subs 2024-03-25T18:12:40.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb778dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:40.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb778dd640 1 -- 172.21.15.81:0/2649214247 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb70140f70 con 0x7fbb70066b50 2024-03-25T18:12:40.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb778dd640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:40.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb778dd640 10 monclient: _renew_subs 2024-03-25T18:12:40.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb778dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:40.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb778dd640 1 -- 172.21.15.81:0/2649214247 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb70137050 con 0x7fbb70066b50 2024-03-25T18:12:40.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.077+0000 7fbb778dd640 1 librados: init done 2024-03-25T18:12:40.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.078+0000 7fbb778dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:40.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.078+0000 7fbb778dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:40.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.078+0000 7fbb778dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:40.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.078+0000 7fbb778dd640 1 -- 172.21.15.81:0/2649214247 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb7006f010 con 0x7fbb70066b50 2024-03-25T18:12:40.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.175+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb6c0038a0 con 0x7fbb70066b50 2024-03-25T18:12:40.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.175+0000 7fbb65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:40.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.175+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fbb6c003bb0 con 0x7fbb70066b50 2024-03-25T18:12:40.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.175+0000 7fbb65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.175+0000 7fbb65ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:40.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.175+0000 7fbb65ffb640 10 monclient: dump: 2024-03-25T18:12:40.177 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.177 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.177 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.177 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.177 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.178 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.178 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.178 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.178 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.178 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.176+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbb6c0040a0 con 0x7fbb70066b50 2024-03-25T18:12:40.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.176+0000 7fbb677fe640 4 set_mon_vals no callback set 2024-03-25T18:12:40.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.176+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3908737589 0 0) 0x7fbb6c005600 con 0x7fbb70066b50 2024-03-25T18:12:40.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.176+0000 7fbb65ffb640 10 client.4138.objecter ms_dispatch 0x7fbb70000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:12:40.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.176+0000 7fbb65ffb640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:12:40.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.176+0000 7fbb65ffb640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:12:40.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.176+0000 7fbb65ffb640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T18:12:40.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.176+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbb6c005a00 con 0x7fbb70066b50 2024-03-25T18:12:40.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.176+0000 7fbb65ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:40.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.176+0000 7fbb65ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:40.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.248+0000 7fbb778dd640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b0b711e9-7872-4700-83a6-986664c998d5", "id": 4}] 2024-03-25T18:12:40.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.248+0000 7fbb778dd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b0b711e9-7872-4700-83a6-986664c998d5", "id": 4}] 2024-03-25T18:12:40.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.248+0000 7fbb778dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:40.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.248+0000 7fbb778dd640 1 -- 172.21.15.81:0/2649214247 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b0b711e9-7872-4700-83a6-986664c998d5", "id": 4} v 0) v1 -- 0x7fbb70141310 con 0x7fbb70066b50 2024-03-25T18:12:40.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.312+0000 7fbb65ffb640 1 -- 172.21.15.81:0/2649214247 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b0b711e9-7872-4700-83a6-986664c998d5", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1954099993 0 1697012583) 0x7fbb6c005ba0 con 0x7fbb70066b50 2024-03-25T18:12:40.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.312+0000 7fbb65ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b0b711e9-7872-4700-83a6-986664c998d5", "id": 4}] 2024-03-25T18:12:40.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.312+0000 7fbb65ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:40.313 INFO:teuthology.orchestra.run.smithi081.stdout:4 2024-03-25T18:12:40.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.313+0000 7fbb778dd640 10 librados: watch_flush enter 2024-03-25T18:12:40.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.313+0000 7fbb778dd640 10 librados: watch_flush exit 2024-03-25T18:12:40.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.313+0000 7fbb778dd640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T18:12:40.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.313+0000 7fbb778dd640 10 client.4138.objecter successfully canceled tick 2024-03-25T18:12:40.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.313+0000 7fbb778dd640 10 monclient: shutdown 2024-03-25T18:12:40.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.313+0000 7fbb778dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:40.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.313+0000 7fbb778dd640 1 -- 172.21.15.81:0/2649214247 >> v1:172.21.15.81:6789/0 conn(0x7fbb70066b50 legacy=0x7fbb70143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.313+0000 7fbb778dd640 1 -- 172.21.15.81:0/2649214247 shutdown_connections 2024-03-25T18:12:40.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.313+0000 7fbb778dd640 1 -- 172.21.15.81:0/2649214247 >> 172.21.15.81:0/2649214247 conn(0x7fbb700b46e0 msgr2=0x7fbb700ae000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:40.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.313+0000 7fbb778dd640 1 -- 172.21.15.81:0/2649214247 shutdown_connections 2024-03-25T18:12:40.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.313+0000 7fbb778dd640 1 -- 172.21.15.81:0/2649214247 wait complete. 2024-03-25T18:12:40.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.313+0000 7fbb778dd640 1 librados: shutdown 2024-03-25T18:12:40.324 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 9c189d47-3570-4000-92d2-c1a733788e62 5 2024-03-25T18:12:40.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.422+0000 7fe604bb5640 10 monclient: get_monmap_and_config 2024-03-25T18:12:40.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.422+0000 7fe604bb5640 10 monclient: build_initial_monmap 2024-03-25T18:12:40.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.422+0000 7fe604bb5640 10 monclient: monmap: 2024-03-25T18:12:40.423 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T18:12:40.423 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.423 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:12:40.423182+0000 2024-03-25T18:12:40.423 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:12:40.423182+0000 2024-03-25T18:12:40.423 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:40.423 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.423 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:40.423 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:12:40.423 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:12:40.423 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.422+0000 7fe604bb5640 1 Processor -- start 2024-03-25T18:12:40.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.422+0000 7fe604bb5640 1 -- start start 2024-03-25T18:12:40.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.422+0000 7fe604bb5640 10 monclient: init 2024-03-25T18:12:40.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.423+0000 7fe604bb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:40.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.423+0000 7fe604bb5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:40.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.423+0000 7fe604bb5640 10 monclient(hunting): picked mon.noname-b con 0x7fe6000a9920 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:40.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.423+0000 7fe604bb5640 10 monclient(hunting): picked mon.noname-c con 0x7fe6000b45a0 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:40.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.423+0000 7fe604bb5640 10 monclient(hunting): picked mon.noname-a con 0x7fe600146450 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:40.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.423+0000 7fe604bb5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60006b100 con 0x7fe600146450 2024-03-25T18:12:40.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.423+0000 7fe604bb5640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6000b2940 con 0x7fe6000b45a0 2024-03-25T18:12:40.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.423+0000 7fe604bb5640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60013fec0 con 0x7fe6000a9920 2024-03-25T18:12:40.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.423+0000 7fe604bb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:40.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.423+0000 7fe604bb5640 10 monclient(hunting): authenticate will time out at 7465.603245s 2024-03-25T18:12:40.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.424+0000 7fe5fdd9b640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe6000b45a0 0x7fe600145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:47344/0 (socket says 172.21.15.81:47344) 2024-03-25T18:12:40.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.424+0000 7fe5fdd9b640 1 -- 172.21.15.81:0/1842667669 learned_addr learned my addr 172.21.15.81:0/1842667669 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T18:12:40.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 1 -- 172.21.15.81:0/1842667669 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15131893 0 0) 0x7fe6000b2940 con 0x7fe6000b45a0 2024-03-25T18:12:40.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:12:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 1 -- 172.21.15.81:0/1842667669 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5e8003520 con 0x7fe6000b45a0 2024-03-25T18:12:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 1 -- 172.21.15.81:0/1842667669 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527301566 0 0) 0x7fe60006b100 con 0x7fe600146450 2024-03-25T18:12:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:12:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 1 -- 172.21.15.81:0/1842667669 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6000b2940 con 0x7fe600146450 2024-03-25T18:12:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 1 -- 172.21.15.81:0/1842667669 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203327932 0 0) 0x7fe60013fec0 con 0x7fe6000a9920 2024-03-25T18:12:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:12:40.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.426+0000 7fe5fd59a640 1 -- 172.21.15.81:0/1842667669 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe60006b100 con 0x7fe6000a9920 2024-03-25T18:12:40.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe5fd59a640 1 -- 172.21.15.81:0/1842667669 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2194170458 0 0) 0x7fe5e8003520 con 0x7fe6000b45a0 2024-03-25T18:12:40.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe5fd59a640 1 -- 172.21.15.81:0/1842667669 >> v1:172.21.15.63:6790/0 conn(0x7fe6000a9920 legacy=0x7fe60013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe5fd59a640 1 -- 172.21.15.81:0/1842667669 >> v1:172.21.15.63:6789/0 conn(0x7fe600146450 legacy=0x7fe60014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe5fd59a640 10 monclient: _finish_hunting 0 2024-03-25T18:12:40.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe5fd59a640 1 monclient: found mon.noname-c 2024-03-25T18:12:40.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe5fd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:40.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe5fd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:12:40.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe5fd59a640 1 -- 172.21.15.81:0/1842667669 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe600140050 con 0x7fe6000b45a0 2024-03-25T18:12:40.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe5fd59a640 10 monclient: _finish_auth 0 2024-03-25T18:12:40.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe5fd59a640 10 monclient: _check_auth_tickets 2024-03-25T18:12:40.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe5fd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:40.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe5fd59a640 1 -- 172.21.15.81:0/1842667669 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe5f4001690 con 0x7fe6000b45a0 2024-03-25T18:12:40.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe604bb5640 5 monclient: authenticate success, global_id 4141 2024-03-25T18:12:40.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe604bb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:40.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe5fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe5fd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:40.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe5fd59a640 10 monclient: dump: 2024-03-25T18:12:40.431 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.431 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.431 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.431 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.432 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.432 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.432 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.432 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.432 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.432 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.429+0000 7fe604bb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:40.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.431+0000 7fe5fd59a640 1 -- 172.21.15.81:0/1842667669 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5f40017f0 con 0x7fe6000b45a0 2024-03-25T18:12:40.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.431+0000 7fe5fd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:40.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.431+0000 7fe5fd59a640 1 -- 172.21.15.81:0/1842667669 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe5f4001a60 con 0x7fe6000b45a0 2024-03-25T18:12:40.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.431+0000 7fe5fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.431+0000 7fe5fd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:40.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.431+0000 7fe5fd59a640 10 monclient: dump: 2024-03-25T18:12:40.433 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.433 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.433 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.433 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.434 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.434 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.434 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.434 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.434 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.434 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.431+0000 7fe604bb5640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:40.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.431+0000 7fe604bb5640 4 set_mon_vals no callback set 2024-03-25T18:12:40.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.432+0000 7fe604bb5640 10 monclient: shutdown 2024-03-25T18:12:40.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.432+0000 7fe604bb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:40.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.432+0000 7fe604bb5640 1 -- 172.21.15.81:0/1842667669 >> v1:172.21.15.81:6789/0 conn(0x7fe6000b45a0 legacy=0x7fe600145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.432+0000 7fe604bb5640 1 -- 172.21.15.81:0/1842667669 shutdown_connections 2024-03-25T18:12:40.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.432+0000 7fe604bb5640 1 -- 172.21.15.81:0/1842667669 >> 172.21.15.81:0/1842667669 conn(0x7fe6000b7f40 msgr2=0x7fe6000b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:40.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.432+0000 7fe604bb5640 1 -- 172.21.15.81:0/1842667669 shutdown_connections 2024-03-25T18:12:40.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.432+0000 7fe604bb5640 1 -- 172.21.15.81:0/1842667669 wait complete. 2024-03-25T18:12:40.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.433+0000 7fe604bb5640 10 monclient: build_initial_monmap 2024-03-25T18:12:40.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.433+0000 7fe604bb5640 10 monclient: monmap: 2024-03-25T18:12:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T18:12:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-25T18:12:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-25T18:12:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-25T18:12:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:12:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:12:40.437 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.433+0000 7fe604bb5640 1 librados: starting msgr at 2024-03-25T18:12:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.433+0000 7fe604bb5640 1 librados: starting objecter 2024-03-25T18:12:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.433+0000 7fe604bb5640 1 Processor -- start 2024-03-25T18:12:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.433+0000 7fe604bb5640 1 -- start start 2024-03-25T18:12:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.433+0000 7fe604bb5640 1 librados: setting wanted keys 2024-03-25T18:12:40.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.433+0000 7fe604bb5640 1 librados: calling monclient init 2024-03-25T18:12:40.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.433+0000 7fe604bb5640 10 monclient: init 2024-03-25T18:12:40.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.433+0000 7fe604bb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:40.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.434+0000 7fe604bb5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:40.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.434+0000 7fe604bb5640 10 monclient(hunting): picked mon.noname-c con 0x7fe6000a9920 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:40.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.434+0000 7fe604bb5640 10 monclient(hunting): picked mon.noname-a con 0x7fe6000b45a0 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:40.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.434+0000 7fe604bb5640 10 monclient(hunting): picked mon.noname-b con 0x7fe600146450 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:40.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.434+0000 7fe604bb5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6000ac0c0 con 0x7fe6000b45a0 2024-03-25T18:12:40.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.434+0000 7fe604bb5640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6000ac330 con 0x7fe600146450 2024-03-25T18:12:40.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.434+0000 7fe604bb5640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6001db220 con 0x7fe6000a9920 2024-03-25T18:12:40.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.434+0000 7fe604bb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:40.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.434+0000 7fe604bb5640 10 monclient(hunting): authenticate will time out at 7465.613741s 2024-03-25T18:12:40.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.434+0000 7fe5fed9d640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe600146450 0x7fe600144810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:47346/0 (socket says 172.21.15.81:47346) 2024-03-25T18:12:40.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.434+0000 7fe5fed9d640 1 -- 172.21.15.81:0/2478580702 learned_addr learned my addr 172.21.15.81:0/2478580702 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T18:12:40.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.434+0000 7fe5eeffd640 10 client.?.objecter ms_handle_connect 0x7fe600146450 2024-03-25T18:12:40.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.434+0000 7fe5eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:40.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.435+0000 7fe5eeffd640 10 client.?.objecter ms_handle_connect 0x7fe6000a9920 2024-03-25T18:12:40.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.435+0000 7fe5eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:40.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.435+0000 7fe5eeffd640 10 client.?.objecter ms_handle_connect 0x7fe6000b45a0 2024-03-25T18:12:40.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.435+0000 7fe5eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:40.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.436+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358093558 0 0) 0x7fe6000ac330 con 0x7fe600146450 2024-03-25T18:12:40.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.436+0000 7fe5eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.436+0000 7fe5eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.436+0000 7fe5eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.436+0000 7fe5eeffd640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:12:40.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.436+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5e0004eb0 con 0x7fe600146450 2024-03-25T18:12:40.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.437+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704599438 0 0) 0x7fe6001db220 con 0x7fe6000a9920 2024-03-25T18:12:40.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.437+0000 7fe5eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.437+0000 7fe5eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.437+0000 7fe5eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.437+0000 7fe5eeffd640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:12:40.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.437+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6000ac330 con 0x7fe6000a9920 2024-03-25T18:12:40.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.437+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041908531 0 0) 0x7fe6000ac0c0 con 0x7fe6000b45a0 2024-03-25T18:12:40.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.437+0000 7fe5eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.437+0000 7fe5eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.437+0000 7fe5eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.437+0000 7fe5eeffd640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:12:40.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.437+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6001db220 con 0x7fe6000b45a0 2024-03-25T18:12:40.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.441+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1453042229 0 0) 0x7fe5e0004eb0 con 0x7fe600146450 2024-03-25T18:12:40.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.441+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6000ac0c0 con 0x7fe600146450 2024-03-25T18:12:40.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.441+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe5f8001c90 con 0x7fe600146450 2024-03-25T18:12:40.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.441+0000 7fe5eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.441+0000 7fe5eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:40.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.441+0000 7fe5eeffd640 10 monclient(hunting): dump: 2024-03-25T18:12:40.442 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.442 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.441+0000 7fe5eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.441+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3768432191 0 0) 0x7fe6001db220 con 0x7fe6000b45a0 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.441+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5e0004eb0 con 0x7fe6000b45a0 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.441+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe5f4001960 con 0x7fe6000b45a0 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.441+0000 7fe5eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.441+0000 7fe5eeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.441+0000 7fe5eeffd640 10 monclient(hunting): dump: 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.444 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.444 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.444 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.444 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.444 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.444 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.444 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.444 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.443+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 894539763 0 0) 0x7fe6000ac330 con 0x7fe6000a9920 2024-03-25T18:12:40.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.443+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6001db220 con 0x7fe6000a9920 2024-03-25T18:12:40.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.443+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe5f0001870 con 0x7fe6000a9920 2024-03-25T18:12:40.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.443+0000 7fe5eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.443+0000 7fe5eeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:40.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.443+0000 7fe5eeffd640 10 monclient(hunting): dump: 2024-03-25T18:12:40.445 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.445 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.445 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.445 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.445 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.445 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.445 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.445 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.445 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.445 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3857357534 0 0) 0x7fe6000ac0c0 con 0x7fe600146450 2024-03-25T18:12:40.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 >> v1:172.21.15.63:6790/0 conn(0x7fe6000a9920 legacy=0x7fe6001da830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 >> v1:172.21.15.63:6789/0 conn(0x7fe6000b45a0 legacy=0x7fe600144100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe5eeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:40.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe5eeffd640 1 monclient: found mon.b 2024-03-25T18:12:40.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe5eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:40.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe5eeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:40.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6001daf40 con 0x7fe600146450 2024-03-25T18:12:40.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe5eeffd640 10 monclient: _finish_auth 0 2024-03-25T18:12:40.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe5eeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:40.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe5eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:40.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe604bb5640 5 monclient: authenticate success, global_id 4144 2024-03-25T18:12:40.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe604bb5640 10 monclient: _renew_subs 2024-03-25T18:12:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe604bb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe604bb5640 1 -- 172.21.15.81:0/2478580702 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6001e2940 con 0x7fe600146450 2024-03-25T18:12:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe604bb5640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe604bb5640 10 monclient: _renew_subs 2024-03-25T18:12:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe604bb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe604bb5640 1 -- 172.21.15.81:0/2478580702 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6001e2cb0 con 0x7fe600146450 2024-03-25T18:12:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.445+0000 7fe604bb5640 1 librados: init done 2024-03-25T18:12:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.446+0000 7fe604bb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.446+0000 7fe604bb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.446+0000 7fe604bb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.446+0000 7fe604bb5640 1 -- 172.21.15.81:0/2478580702 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60004f730 con 0x7fe600146450 2024-03-25T18:12:40.453 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:40.453+0000 7f0c0da1f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:12:40.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.545+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5f8003970 con 0x7fe600146450 2024-03-25T18:12:40.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.545+0000 7fe5eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:40.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.545+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe5f8003d80 con 0x7fe600146450 2024-03-25T18:12:40.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.545+0000 7fe5eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.545+0000 7fe5eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:40.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.545+0000 7fe5eeffd640 10 monclient: dump: 2024-03-25T18:12:40.546 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.546 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.546 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.546 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.547 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.547 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.547 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.547 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.547 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.547 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.545+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe5f8004330 con 0x7fe600146450 2024-03-25T18:12:40.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.545+0000 7fe5fcd99640 4 set_mon_vals no callback set 2024-03-25T18:12:40.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.545+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1358615041 0 0) 0x7fe5f8006a80 con 0x7fe600146450 2024-03-25T18:12:40.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.545+0000 7fe5eeffd640 10 client.4144.objecter ms_dispatch 0x7fe600000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:12:40.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.545+0000 7fe5eeffd640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:12:40.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.545+0000 7fe5eeffd640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:12:40.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.545+0000 7fe5eeffd640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T18:12:40.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.545+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe5f8018de0 con 0x7fe600146450 2024-03-25T18:12:40.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.545+0000 7fe5eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:40.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.545+0000 7fe5eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:40.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.618+0000 7fe604bb5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9c189d47-3570-4000-92d2-c1a733788e62", "id": 5}] 2024-03-25T18:12:40.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.618+0000 7fe604bb5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9c189d47-3570-4000-92d2-c1a733788e62", "id": 5}] 2024-03-25T18:12:40.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.618+0000 7fe604bb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:40.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.618+0000 7fe604bb5640 1 -- 172.21.15.81:0/2478580702 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9c189d47-3570-4000-92d2-c1a733788e62", "id": 5} v 0) v1 -- 0x7fe60013b4d0 con 0x7fe600146450 2024-03-25T18:12:40.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.696+0000 7fe5eeffd640 1 -- 172.21.15.81:0/2478580702 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9c189d47-3570-4000-92d2-c1a733788e62", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1070738263 0 1988412176) 0x7fe5f8006500 con 0x7fe600146450 2024-03-25T18:12:40.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.696+0000 7fe5eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9c189d47-3570-4000-92d2-c1a733788e62", "id": 5}] 2024-03-25T18:12:40.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.696+0000 7fe5eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:40.697 INFO:teuthology.orchestra.run.smithi081.stdout:5 2024-03-25T18:12:40.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.697+0000 7fe604bb5640 10 librados: watch_flush enter 2024-03-25T18:12:40.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.697+0000 7fe604bb5640 10 librados: watch_flush exit 2024-03-25T18:12:40.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.697+0000 7fe604bb5640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T18:12:40.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.697+0000 7fe604bb5640 10 client.4144.objecter successfully canceled tick 2024-03-25T18:12:40.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.697+0000 7fe604bb5640 10 monclient: shutdown 2024-03-25T18:12:40.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.697+0000 7fe604bb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:40.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.697+0000 7fe604bb5640 1 -- 172.21.15.81:0/2478580702 >> v1:172.21.15.81:6789/0 conn(0x7fe600146450 legacy=0x7fe600144810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.697+0000 7fe604bb5640 1 -- 172.21.15.81:0/2478580702 shutdown_connections 2024-03-25T18:12:40.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.697+0000 7fe604bb5640 1 -- 172.21.15.81:0/2478580702 >> 172.21.15.81:0/2478580702 conn(0x7fe6000b7f40 msgr2=0x7fe600074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:40.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.697+0000 7fe604bb5640 1 -- 172.21.15.81:0/2478580702 shutdown_connections 2024-03-25T18:12:40.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.697+0000 7fe604bb5640 1 -- 172.21.15.81:0/2478580702 wait complete. 2024-03-25T18:12:40.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.697+0000 7fe604bb5640 1 librados: shutdown 2024-03-25T18:12:40.708 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 4b8938e4-7c72-453d-bed9-df8bce519abb 6 2024-03-25T18:12:40.738 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:40.737+0000 7f0c0da1f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:12:40.755 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:40.754+0000 7fd9c8010040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:12:40.794 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:40.793+0000 7f0c0da1f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:12:40.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.809+0000 7fe9472c8640 10 monclient: get_monmap_and_config 2024-03-25T18:12:40.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.809+0000 7fe9472c8640 10 monclient: build_initial_monmap 2024-03-25T18:12:40.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.809+0000 7fe9472c8640 10 monclient: monmap: 2024-03-25T18:12:40.810 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T18:12:40.810 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.810 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:12:40.810732+0000 2024-03-25T18:12:40.811 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:12:40.810732+0000 2024-03-25T18:12:40.811 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:40.811 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.811 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:40.811 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:12:40.811 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:12:40.811 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.810+0000 7fe9472c8640 1 Processor -- start 2024-03-25T18:12:40.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.810+0000 7fe9472c8640 1 -- start start 2024-03-25T18:12:40.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.810+0000 7fe9472c8640 10 monclient: init 2024-03-25T18:12:40.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.811+0000 7fe9472c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:40.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.811+0000 7fe9472c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:40.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.811+0000 7fe9472c8640 10 monclient(hunting): picked mon.noname-b con 0x7fe9400b4700 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:40.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.811+0000 7fe9472c8640 10 monclient(hunting): picked mon.noname-a con 0x7fe940142a20 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:40.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.811+0000 7fe9472c8640 10 monclient(hunting): picked mon.noname-c con 0x7fe9400a96b0 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:40.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.811+0000 7fe9472c8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9400a9d60 con 0x7fe940142a20 2024-03-25T18:12:40.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.811+0000 7fe9472c8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94013ff10 con 0x7fe9400a96b0 2024-03-25T18:12:40.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.811+0000 7fe9472c8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9400aa1f0 con 0x7fe9400b4700 2024-03-25T18:12:40.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.811+0000 7fe9472c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:40.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.811+0000 7fe9472c8640 10 monclient(hunting): authenticate will time out at 7465.990760s 2024-03-25T18:12:40.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.811+0000 7fe945064640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fe9400b4700 0x7fe940142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.81:48270/0 (socket says 172.21.15.81:48270) 2024-03-25T18:12:40.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.811+0000 7fe945064640 1 -- 172.21.15.81:0/1217632092 learned_addr learned my addr 172.21.15.81:0/1217632092 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T18:12:40.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.813+0000 7fe937fff640 1 -- 172.21.15.81:0/1217632092 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921975184 0 0) 0x7fe9400aa1f0 con 0x7fe9400b4700 2024-03-25T18:12:40.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.813+0000 7fe937fff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.813+0000 7fe937fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.813+0000 7fe937fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.813+0000 7fe937fff640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:12:40.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.813+0000 7fe937fff640 1 -- 172.21.15.81:0/1217632092 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe92c003520 con 0x7fe9400b4700 2024-03-25T18:12:40.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.814+0000 7fe937fff640 1 -- 172.21.15.81:0/1217632092 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404247172 0 0) 0x7fe94013ff10 con 0x7fe9400a96b0 2024-03-25T18:12:40.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.814+0000 7fe937fff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.814+0000 7fe937fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.814+0000 7fe937fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.814+0000 7fe937fff640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:12:40.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.814+0000 7fe937fff640 1 -- 172.21.15.81:0/1217632092 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9400aa1f0 con 0x7fe9400a96b0 2024-03-25T18:12:40.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.814+0000 7fe937fff640 1 -- 172.21.15.81:0/1217632092 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323235262 0 0) 0x7fe9400a9d60 con 0x7fe940142a20 2024-03-25T18:12:40.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.814+0000 7fe937fff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.814+0000 7fe937fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.814+0000 7fe937fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.814+0000 7fe937fff640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:12:40.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.814+0000 7fe937fff640 1 -- 172.21.15.81:0/1217632092 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe94013ff10 con 0x7fe940142a20 2024-03-25T18:12:40.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe937fff640 1 -- 172.21.15.81:0/1217632092 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2788180526 0 0) 0x7fe92c003520 con 0x7fe9400b4700 2024-03-25T18:12:40.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe937fff640 1 -- 172.21.15.81:0/1217632092 >> v1:172.21.15.81:6789/0 conn(0x7fe9400a96b0 legacy=0x7fe94013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe937fff640 1 -- 172.21.15.81:0/1217632092 >> v1:172.21.15.63:6789/0 conn(0x7fe940142a20 legacy=0x7fe94013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe937fff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:40.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe937fff640 1 monclient: found mon.noname-b 2024-03-25T18:12:40.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe937fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:40.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe937fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:12:40.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe937fff640 1 -- 172.21.15.81:0/1217632092 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9400b4ae0 con 0x7fe9400b4700 2024-03-25T18:12:40.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe937fff640 10 monclient: _finish_auth 0 2024-03-25T18:12:40.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe937fff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:40.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe937fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:40.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe937fff640 1 -- 172.21.15.81:0/1217632092 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe93c001a10 con 0x7fe9400b4700 2024-03-25T18:12:40.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe937fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe937fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:40.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe937fff640 10 monclient: dump: 2024-03-25T18:12:40.819 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.819 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.819 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.819 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.819 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.819 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.819 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.819 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.819 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.820 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe9472c8640 5 monclient: authenticate success, global_id 4151 2024-03-25T18:12:40.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.817+0000 7fe9472c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:40.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.818+0000 7fe937fff640 1 -- 172.21.15.81:0/1217632092 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe93c001b70 con 0x7fe9400b4700 2024-03-25T18:12:40.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.818+0000 7fe937fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:40.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.818+0000 7fe9472c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:40.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.818+0000 7fe9472c8640 4 set_mon_vals no callback set 2024-03-25T18:12:40.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.819+0000 7fe937fff640 1 -- 172.21.15.81:0/1217632092 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe93c001f80 con 0x7fe9400b4700 2024-03-25T18:12:40.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.819+0000 7fe937fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.819+0000 7fe937fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:40.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.819+0000 7fe937fff640 10 monclient: dump: 2024-03-25T18:12:40.822 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.822 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.822 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.822 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.822 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.822 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.822 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.822 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.822 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.822 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.819+0000 7fe9472c8640 10 monclient: shutdown 2024-03-25T18:12:40.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.819+0000 7fe9472c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:40.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.819+0000 7fe9472c8640 1 -- 172.21.15.81:0/1217632092 >> v1:172.21.15.63:6790/0 conn(0x7fe9400b4700 legacy=0x7fe940142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.819+0000 7fe9472c8640 1 -- 172.21.15.81:0/1217632092 shutdown_connections 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.819+0000 7fe9472c8640 1 -- 172.21.15.81:0/1217632092 >> 172.21.15.81:0/1217632092 conn(0x7fe9400668e0 msgr2=0x7fe940066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.819+0000 7fe9472c8640 1 -- 172.21.15.81:0/1217632092 shutdown_connections 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.819+0000 7fe9472c8640 1 -- 172.21.15.81:0/1217632092 wait complete. 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.820+0000 7fe9472c8640 10 monclient: build_initial_monmap 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.820+0000 7fe9472c8640 10 monclient: monmap: 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.820+0000 7fe9472c8640 1 librados: starting msgr at 2024-03-25T18:12:40.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.820+0000 7fe9472c8640 1 librados: starting objecter 2024-03-25T18:12:40.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.820+0000 7fe9472c8640 1 Processor -- start 2024-03-25T18:12:40.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.820+0000 7fe9472c8640 1 -- start start 2024-03-25T18:12:40.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.820+0000 7fe9472c8640 1 librados: setting wanted keys 2024-03-25T18:12:40.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.820+0000 7fe9472c8640 1 librados: calling monclient init 2024-03-25T18:12:40.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.820+0000 7fe9472c8640 10 monclient: init 2024-03-25T18:12:40.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.821+0000 7fe9472c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:40.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.821+0000 7fe9472c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:40.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.821+0000 7fe9472c8640 10 monclient(hunting): picked mon.noname-a con 0x7fe9400a96b0 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:40.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.821+0000 7fe9472c8640 10 monclient(hunting): picked mon.noname-c con 0x7fe9400b4700 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:40.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.821+0000 7fe9472c8640 10 monclient(hunting): picked mon.noname-b con 0x7fe940142a20 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:40.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.821+0000 7fe9472c8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9400ae220 con 0x7fe9400a96b0 2024-03-25T18:12:40.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.821+0000 7fe9472c8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9400ad720 con 0x7fe940142a20 2024-03-25T18:12:40.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.821+0000 7fe9472c8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9400ae450 con 0x7fe9400b4700 2024-03-25T18:12:40.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.821+0000 7fe9472c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.821+0000 7fe9472c8640 10 monclient(hunting): authenticate will time out at 7466.001025s 2024-03-25T18:12:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.821+0000 7fe944863640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fe9400b4700 0x7fe940136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.81:48286/0 (socket says 172.21.15.81:48286) 2024-03-25T18:12:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.821+0000 7fe944863640 1 -- 172.21.15.81:0/3809004203 learned_addr learned my addr 172.21.15.81:0/3809004203 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T18:12:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.822+0000 7fe935ffb640 10 client.?.objecter ms_handle_connect 0x7fe940142a20 2024-03-25T18:12:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.822+0000 7fe935ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.822+0000 7fe935ffb640 10 client.?.objecter ms_handle_connect 0x7fe9400b4700 2024-03-25T18:12:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.822+0000 7fe935ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.822+0000 7fe935ffb640 10 client.?.objecter ms_handle_connect 0x7fe9400a96b0 2024-03-25T18:12:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.822+0000 7fe935ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100090045 0 0) 0x7fe9400ae450 con 0x7fe9400b4700 2024-03-25T18:12:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:12:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe924004eb0 con 0x7fe9400b4700 2024-03-25T18:12:40.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256632154 0 0) 0x7fe9400ad720 con 0x7fe940142a20 2024-03-25T18:12:40.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:12:40.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9400ae450 con 0x7fe940142a20 2024-03-25T18:12:40.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891743624 0 0) 0x7fe9400ae220 con 0x7fe9400a96b0 2024-03-25T18:12:40.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:12:40.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.824+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9400ad720 con 0x7fe9400a96b0 2024-03-25T18:12:40.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.829+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1231461829 0 0) 0x7fe924004eb0 con 0x7fe9400b4700 2024-03-25T18:12:40.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.829+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9400ae220 con 0x7fe9400b4700 2024-03-25T18:12:40.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.829+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe930001970 con 0x7fe9400b4700 2024-03-25T18:12:40.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.829+0000 7fe935ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.829+0000 7fe935ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:40.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.829+0000 7fe935ffb640 10 monclient(hunting): dump: 2024-03-25T18:12:40.830 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.830 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.830 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.830 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.830 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.830 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.830 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.830 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.830 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.830 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.829+0000 7fe935ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:40.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.829+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2233707614 0 0) 0x7fe9400ad720 con 0x7fe9400a96b0 2024-03-25T18:12:40.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.829+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe924004eb0 con 0x7fe9400a96b0 2024-03-25T18:12:40.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.829+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe93c001e80 con 0x7fe9400a96b0 2024-03-25T18:12:40.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.829+0000 7fe935ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.829+0000 7fe935ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.829+0000 7fe935ffb640 10 monclient(hunting): dump: 2024-03-25T18:12:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.832 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.830+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3266039411 0 0) 0x7fe9400ae450 con 0x7fe940142a20 2024-03-25T18:12:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.830+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9400ad720 con 0x7fe940142a20 2024-03-25T18:12:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.830+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe938001930 con 0x7fe940142a20 2024-03-25T18:12:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.830+0000 7fe935ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.830+0000 7fe935ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:40.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.830+0000 7fe935ffb640 10 monclient(hunting): dump: 2024-03-25T18:12:40.833 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.833 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.833 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.833 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.833 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.833 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.833 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.833 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.833 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.833 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.832+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2833361219 0 0) 0x7fe9400ae220 con 0x7fe9400b4700 2024-03-25T18:12:40.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.832+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 >> v1:172.21.15.81:6789/0 conn(0x7fe940142a20 legacy=0x7fe9401e3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.832+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 >> v1:172.21.15.63:6789/0 conn(0x7fe9400a96b0 legacy=0x7fe9400ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.832+0000 7fe935ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:12:40.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.832+0000 7fe935ffb640 1 monclient: found mon.c 2024-03-25T18:12:40.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.832+0000 7fe935ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:40.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.832+0000 7fe935ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:40.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.832+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9400ae730 con 0x7fe9400b4700 2024-03-25T18:12:40.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.832+0000 7fe935ffb640 10 monclient: _finish_auth 0 2024-03-25T18:12:40.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.832+0000 7fe935ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:12:40.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.832+0000 7fe935ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:40.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.832+0000 7fe9472c8640 5 monclient: authenticate success, global_id 4154 2024-03-25T18:12:40.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.832+0000 7fe9472c8640 10 monclient: _renew_subs 2024-03-25T18:12:40.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.832+0000 7fe9472c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:40.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.832+0000 7fe9472c8640 1 -- 172.21.15.81:0/3809004203 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9400b0db0 con 0x7fe9400b4700 2024-03-25T18:12:40.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.833+0000 7fe9472c8640 10 client.4154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:40.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.833+0000 7fe9472c8640 10 monclient: _renew_subs 2024-03-25T18:12:40.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.833+0000 7fe9472c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:40.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.833+0000 7fe9472c8640 1 -- 172.21.15.81:0/3809004203 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94013b490 con 0x7fe9400b4700 2024-03-25T18:12:40.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.833+0000 7fe9472c8640 1 librados: init done 2024-03-25T18:12:40.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.833+0000 7fe9472c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:40.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.833+0000 7fe9472c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:40.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.833+0000 7fe9472c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:40.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.833+0000 7fe9472c8640 1 -- 172.21.15.81:0/3809004203 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9400b1050 con 0x7fe9400b4700 2024-03-25T18:12:40.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.837+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe930003610 con 0x7fe9400b4700 2024-03-25T18:12:40.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.837+0000 7fe935ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:40.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.837+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe930003800 con 0x7fe9400b4700 2024-03-25T18:12:40.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.837+0000 7fe935ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.837+0000 7fe935ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:40.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.837+0000 7fe935ffb640 10 monclient: dump: 2024-03-25T18:12:40.838 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:40.838 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:40.838 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.838 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:40.839 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.839 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:40.839 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:40.839 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:40.839 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:40.839 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:40.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.837+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe930003db0 con 0x7fe9400b4700 2024-03-25T18:12:40.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.837+0000 7fe9377fe640 4 set_mon_vals no callback set 2024-03-25T18:12:40.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.837+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 4175540760 0 0) 0x7fe930006630 con 0x7fe9400b4700 2024-03-25T18:12:40.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.837+0000 7fe935ffb640 10 client.4154.objecter ms_dispatch 0x7fe940000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:12:40.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.837+0000 7fe935ffb640 3 client.4154.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:12:40.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.837+0000 7fe935ffb640 3 client.4154.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:12:40.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.837+0000 7fe935ffb640 20 client.4154.objecter dump_active .. 0 homeless 2024-03-25T18:12:40.915 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:40.914+0000 7f0c0da1f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:12:40.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.937+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe930006150 con 0x7fe9400b4700 2024-03-25T18:12:40.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.937+0000 7fe935ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:40.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:40.937+0000 7fe935ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:41.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.007+0000 7fe9472c8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4b8938e4-7c72-453d-bed9-df8bce519abb", "id": 6}] 2024-03-25T18:12:41.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.007+0000 7fe9472c8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4b8938e4-7c72-453d-bed9-df8bce519abb", "id": 6}] 2024-03-25T18:12:41.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.007+0000 7fe9472c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:41.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.007+0000 7fe9472c8640 1 -- 172.21.15.81:0/3809004203 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd new", "uuid": "4b8938e4-7c72-453d-bed9-df8bce519abb", "id": 6} v 0) v1 -- 0x7fe94004f730 con 0x7fe9400b4700 2024-03-25T18:12:41.050 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:41.050+0000 7fd9c8010040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:12:41.105 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:41.104+0000 7fd9c8010040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:12:41.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.223+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3106638272 0 0) 0x7fe930004060 con 0x7fe9400b4700 2024-03-25T18:12:41.229 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T18:12:41.229+0000 7fd9c8010040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:12:41.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.492+0000 7fe935ffb640 1 -- 172.21.15.81:0/3809004203 <== mon.2 v1:172.21.15.63:6790/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4b8938e4-7c72-453d-bed9-df8bce519abb", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1397010933 0 1113811849) 0x7fe9300063c0 con 0x7fe9400b4700 2024-03-25T18:12:41.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.492+0000 7fe935ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4b8938e4-7c72-453d-bed9-df8bce519abb", "id": 6}] 2024-03-25T18:12:41.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.492+0000 7fe935ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:41.494 INFO:teuthology.orchestra.run.smithi081.stdout:6 2024-03-25T18:12:41.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.493+0000 7fe9472c8640 10 librados: watch_flush enter 2024-03-25T18:12:41.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.493+0000 7fe9472c8640 10 librados: watch_flush exit 2024-03-25T18:12:41.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.493+0000 7fe9472c8640 20 client.4154.objecter shutdown clearing up homeless session... 2024-03-25T18:12:41.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.493+0000 7fe9472c8640 10 client.4154.objecter successfully canceled tick 2024-03-25T18:12:41.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.493+0000 7fe9472c8640 10 monclient: shutdown 2024-03-25T18:12:41.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.493+0000 7fe9472c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:41.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.493+0000 7fe9472c8640 1 -- 172.21.15.81:0/3809004203 >> v1:172.21.15.63:6790/0 conn(0x7fe9400b4700 legacy=0x7fe940136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:41.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.493+0000 7fe9472c8640 1 -- 172.21.15.81:0/3809004203 shutdown_connections 2024-03-25T18:12:41.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.493+0000 7fe9472c8640 1 -- 172.21.15.81:0/3809004203 >> 172.21.15.81:0/3809004203 conn(0x7fe9400668e0 msgr2=0x7fe9400b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:41.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.493+0000 7fe9472c8640 1 -- 172.21.15.81:0/3809004203 shutdown_connections 2024-03-25T18:12:41.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.493+0000 7fe9472c8640 1 -- 172.21.15.81:0/3809004203 wait complete. 2024-03-25T18:12:41.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.493+0000 7fe9472c8640 1 librados: shutdown 2024-03-25T18:12:41.504 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 1c628d65-bba1-4a42-bc80-dc5197e22e2b 7 2024-03-25T18:12:41.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.643+0000 7f62c6932640 10 monclient: get_monmap_and_config 2024-03-25T18:12:41.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.643+0000 7f62c6932640 10 monclient: build_initial_monmap 2024-03-25T18:12:41.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.643+0000 7f62c6932640 10 monclient: monmap: 2024-03-25T18:12:41.644 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T18:12:41.644 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:41.644 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:12:41.644609+0000 2024-03-25T18:12:41.644 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:12:41.644609+0000 2024-03-25T18:12:41.644 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:41.645 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:41.645 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:41.645 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:12:41.645 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:12:41.645 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:41.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.644+0000 7f62c6932640 1 Processor -- start 2024-03-25T18:12:41.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.644+0000 7f62c6932640 1 -- start start 2024-03-25T18:12:41.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.644+0000 7f62c6932640 10 monclient: init 2024-03-25T18:12:41.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.645+0000 7f62c6932640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:41.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.645+0000 7f62c6932640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:41.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.645+0000 7f62c6932640 10 monclient(hunting): picked mon.noname-c con 0x7f62c006aba0 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:41.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.645+0000 7f62c6932640 10 monclient(hunting): picked mon.noname-a con 0x7f62c006af80 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:41.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.645+0000 7f62c6932640 10 monclient(hunting): picked mon.noname-b con 0x7f62c0072600 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:41.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.645+0000 7f62c6932640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c013fff0 con 0x7f62c006af80 2024-03-25T18:12:41.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.645+0000 7f62c6932640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c00afd80 con 0x7f62c006aba0 2024-03-25T18:12:41.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.645+0000 7f62c6932640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c00b11f0 con 0x7f62c0072600 2024-03-25T18:12:41.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.645+0000 7f62c6932640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:41.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.645+0000 7f62c6932640 10 monclient(hunting): authenticate will time out at 7466.824748s 2024-03-25T18:12:41.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.645+0000 7f62c4ecf640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f62c0072600 0x7f62c0142380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.81:48360/0 (socket says 172.21.15.81:48360) 2024-03-25T18:12:41.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.645+0000 7f62c4ecf640 1 -- 172.21.15.81:0/3737847063 learned_addr learned my addr 172.21.15.81:0/3737847063 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T18:12:41.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.682+0000 7f62beffd640 1 -- 172.21.15.81:0/3737847063 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567612949 0 0) 0x7f62c013fff0 con 0x7f62c006af80 2024-03-25T18:12:41.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.683+0000 7f62beffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:41.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.683+0000 7f62beffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:41.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.683+0000 7f62beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:41.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.683+0000 7f62beffd640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:12:41.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.683+0000 7f62beffd640 1 -- 172.21.15.81:0/3737847063 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62a8003520 con 0x7f62c006af80 2024-03-25T18:12:41.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62beffd640 1 -- 172.21.15.81:0/3737847063 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3717614460 0 0) 0x7f62a8003520 con 0x7f62c006af80 2024-03-25T18:12:41.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62beffd640 1 -- 172.21.15.81:0/3737847063 >> v1:172.21.15.63:6790/0 conn(0x7f62c0072600 legacy=0x7f62c0142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:41.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62beffd640 1 -- 172.21.15.81:0/3737847063 >> v1:172.21.15.81:6789/0 conn(0x7f62c006aba0 legacy=0x7f62c00af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:41.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62beffd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:41.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62beffd640 1 monclient: found mon.noname-a 2024-03-25T18:12:41.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:41.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62beffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:12:41.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62beffd640 1 -- 172.21.15.81:0/3737847063 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c00aeff0 con 0x7f62c006af80 2024-03-25T18:12:41.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62beffd640 10 monclient: _finish_auth 0 2024-03-25T18:12:41.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62beffd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:41.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:41.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62beffd640 1 -- 172.21.15.81:0/3737847063 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f62b4001690 con 0x7f62c006af80 2024-03-25T18:12:41.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:41.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:41.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62beffd640 10 monclient: dump: 2024-03-25T18:12:41.688 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:41.688 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:41.688 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:41.688 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:41.688 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:41.688 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:41.688 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:41.688 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:41.688 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:41.688 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:41.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62c6932640 5 monclient: authenticate success, global_id 4158 2024-03-25T18:12:41.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.686+0000 7f62c6932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:41.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.688+0000 7f62beffd640 1 -- 172.21.15.81:0/3737847063 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62b40017f0 con 0x7f62c006af80 2024-03-25T18:12:41.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.688+0000 7f62beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:41.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.688+0000 7f62beffd640 1 -- 172.21.15.81:0/3737847063 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f62b4001c00 con 0x7f62c006af80 2024-03-25T18:12:41.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.688+0000 7f62beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:41.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.688+0000 7f62beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:41.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.688+0000 7f62beffd640 10 monclient: dump: 2024-03-25T18:12:41.690 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:41.690 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:41.690 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:41.690 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:41.690 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:41.690 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:41.690 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:41.691 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:41.691 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:41.691 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:41.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.688+0000 7f62c6932640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:41.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.688+0000 7f62c6932640 4 set_mon_vals no callback set 2024-03-25T18:12:41.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.688+0000 7f62c6932640 10 monclient: shutdown 2024-03-25T18:12:41.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.688+0000 7f62c6932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:41.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.688+0000 7f62c6932640 1 -- 172.21.15.81:0/3737847063 >> v1:172.21.15.63:6789/0 conn(0x7f62c006af80 legacy=0x7f62c00718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:41.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.689+0000 7f62c6932640 1 -- 172.21.15.81:0/3737847063 shutdown_connections 2024-03-25T18:12:41.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.689+0000 7f62c6932640 1 -- 172.21.15.81:0/3737847063 >> 172.21.15.81:0/3737847063 conn(0x7f62c00b4bb0 msgr2=0x7f62c00b4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:41.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.689+0000 7f62c6932640 1 -- 172.21.15.81:0/3737847063 shutdown_connections 2024-03-25T18:12:41.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.689+0000 7f62c6932640 1 -- 172.21.15.81:0/3737847063 wait complete. 2024-03-25T18:12:41.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.689+0000 7f62c6932640 10 monclient: build_initial_monmap 2024-03-25T18:12:41.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.689+0000 7f62c6932640 10 monclient: monmap: 2024-03-25T18:12:41.691 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T18:12:41.692 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:41.692 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-25T18:12:41.692 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-25T18:12:41.692 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:41.692 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-25T18:12:41.692 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:41.692 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:12:41.692 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:12:41.692 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:41.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.689+0000 7f62c6932640 1 librados: starting msgr at 2024-03-25T18:12:41.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.689+0000 7f62c6932640 1 librados: starting objecter 2024-03-25T18:12:41.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.690+0000 7f62c6932640 1 Processor -- start 2024-03-25T18:12:41.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.690+0000 7f62c6932640 1 -- start start 2024-03-25T18:12:41.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.690+0000 7f62c6932640 1 librados: setting wanted keys 2024-03-25T18:12:41.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.690+0000 7f62c6932640 1 librados: calling monclient init 2024-03-25T18:12:41.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.690+0000 7f62c6932640 10 monclient: init 2024-03-25T18:12:41.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.690+0000 7f62c6932640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:41.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.690+0000 7f62c6932640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:41.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.690+0000 7f62c6932640 10 monclient(hunting): picked mon.noname-c con 0x7f62c006aba0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:41.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.690+0000 7f62c6932640 10 monclient(hunting): picked mon.noname-b con 0x7f62c0072600 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:41.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.690+0000 7f62c6932640 10 monclient(hunting): picked mon.noname-a con 0x7f62c013d120 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:41.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.690+0000 7f62c6932640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c0144b80 con 0x7f62c013d120 2024-03-25T18:12:41.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.690+0000 7f62c6932640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c01434b0 con 0x7f62c0072600 2024-03-25T18:12:41.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.690+0000 7f62c6932640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c0145f40 con 0x7f62c006aba0 2024-03-25T18:12:41.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.690+0000 7f62c6932640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:41.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.690+0000 7f62c6932640 10 monclient(hunting): authenticate will time out at 7466.870442s 2024-03-25T18:12:41.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.691+0000 7f62c4ecf640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f62c013d120 0x7f62c01e2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.81:34442/0 (socket says 172.21.15.81:34442) 2024-03-25T18:12:41.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.691+0000 7f62c4ecf640 1 -- 172.21.15.81:0/90656277 learned_addr learned my addr 172.21.15.81:0/90656277 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T18:12:41.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.691+0000 7f62bcff9640 10 client.?.objecter ms_handle_connect 0x7f62c013d120 2024-03-25T18:12:41.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.691+0000 7f62bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:41.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.693+0000 7f62bcff9640 1 -- 172.21.15.81:0/90656277 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121853127 0 0) 0x7f62c0144b80 con 0x7f62c013d120 2024-03-25T18:12:41.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.693+0000 7f62bcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:41.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.693+0000 7f62bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:41.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.693+0000 7f62bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:41.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.693+0000 7f62bcff9640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:12:41.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.693+0000 7f62bcff9640 1 -- 172.21.15.81:0/90656277 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62ac004eb0 con 0x7f62c013d120 2024-03-25T18:12:41.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.698+0000 7f62bcff9640 1 -- 172.21.15.81:0/90656277 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1876544628 0 0) 0x7f62ac004eb0 con 0x7f62c013d120 2024-03-25T18:12:41.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.698+0000 7f62bcff9640 1 -- 172.21.15.81:0/90656277 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62c0144b80 con 0x7f62c013d120 2024-03-25T18:12:41.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.698+0000 7f62bcff9640 1 -- 172.21.15.81:0/90656277 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f62b8001b10 con 0x7f62c013d120 2024-03-25T18:12:41.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.698+0000 7f62bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:41.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.698+0000 7f62bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:41.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.698+0000 7f62bcff9640 10 monclient(hunting): dump: 2024-03-25T18:12:41.700 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:41.700 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:41.700 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:41.700 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:41.700 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:41.700 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:41.700 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:41.700 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:41.700 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:41.700 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:41.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.698+0000 7f62bcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:41.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62bcff9640 1 -- 172.21.15.81:0/90656277 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3685239310 0 0) 0x7f62c0144b80 con 0x7f62c013d120 2024-03-25T18:12:41.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62bcff9640 1 -- 172.21.15.81:0/90656277 >> v1:172.21.15.63:6790/0 conn(0x7f62c006aba0 legacy=0x7f62c0143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:41.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62bcff9640 1 -- 172.21.15.81:0/90656277 >> v1:172.21.15.81:6789/0 conn(0x7f62c0072600 legacy=0x7f62c013ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62bcff9640 10 monclient: _finish_hunting 0 2024-03-25T18:12:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62bcff9640 1 monclient: found mon.a 2024-03-25T18:12:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62bcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:12:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62bcff9640 1 -- 172.21.15.81:0/90656277 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c0145030 con 0x7f62c013d120 2024-03-25T18:12:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62bcff9640 10 monclient: _finish_auth 0 2024-03-25T18:12:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62bcff9640 10 monclient: _check_auth_tickets 2024-03-25T18:12:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62c6932640 5 monclient: authenticate success, global_id 4161 2024-03-25T18:12:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62c6932640 10 monclient: _renew_subs 2024-03-25T18:12:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62c6932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:12:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62c6932640 1 -- 172.21.15.81:0/90656277 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c0144580 con 0x7f62c013d120 2024-03-25T18:12:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62c6932640 10 client.4161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62c6932640 10 monclient: _renew_subs 2024-03-25T18:12:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62c6932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:12:41.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62c6932640 1 -- 172.21.15.81:0/90656277 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c01e3c60 con 0x7f62c013d120 2024-03-25T18:12:41.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.701+0000 7f62c6932640 1 librados: init done 2024-03-25T18:12:41.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.702+0000 7f62c6932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:41.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.702+0000 7f62c6932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:41.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.702+0000 7f62c6932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:12:41.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.702+0000 7f62c6932640 1 -- 172.21.15.81:0/90656277 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c0067f70 con 0x7f62c013d120 2024-03-25T18:12:41.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.815+0000 7f62bcff9640 1 -- 172.21.15.81:0/90656277 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62b80020c0 con 0x7f62c013d120 2024-03-25T18:12:41.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.815+0000 7f62bcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.815+0000 7f62bcff9640 1 -- 172.21.15.81:0/90656277 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f62b8004d00 con 0x7f62c013d120 2024-03-25T18:12:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.815+0000 7f62bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.815+0000 7f62bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.815+0000 7f62bcff9640 10 monclient: dump: 2024-03-25T18:12:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T18:12:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T18:12:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:41.816 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:12:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.815+0000 7f62bdffb640 4 set_mon_vals no callback set 2024-03-25T18:12:41.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.815+0000 7f62bcff9640 1 -- 172.21.15.81:0/90656277 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3106638272 0 0) 0x7f62b8004f20 con 0x7f62c013d120 2024-03-25T18:12:41.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.817+0000 7f62bcff9640 1 -- 172.21.15.81:0/90656277 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 663567326 0 0) 0x7f62b8015a70 con 0x7f62c013d120 2024-03-25T18:12:41.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.817+0000 7f62bcff9640 10 client.4161.objecter ms_dispatch 0x7f62c0000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:12:41.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.817+0000 7f62bcff9640 3 client.4161.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:12:41.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.817+0000 7f62bcff9640 3 client.4161.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:12:41.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.817+0000 7f62bcff9640 20 client.4161.objecter dump_active .. 0 homeless 2024-03-25T18:12:41.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.817+0000 7f62bcff9640 1 -- 172.21.15.81:0/90656277 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f62b8014d30 con 0x7f62c013d120 2024-03-25T18:12:41.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.817+0000 7f62bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:41.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.817+0000 7f62bcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:41.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.893+0000 7f62c6932640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1c628d65-bba1-4a42-bc80-dc5197e22e2b", "id": 7}] 2024-03-25T18:12:41.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.893+0000 7f62c6932640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1c628d65-bba1-4a42-bc80-dc5197e22e2b", "id": 7}] 2024-03-25T18:12:41.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.894+0000 7f62c6932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:12:41.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:41.894+0000 7f62c6932640 1 -- 172.21.15.81:0/90656277 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1c628d65-bba1-4a42-bc80-dc5197e22e2b", "id": 7} v 0) v1 -- 0x7f62c0146460 con 0x7f62c013d120 2024-03-25T18:12:42.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:42.152+0000 7f62bcff9640 1 -- 172.21.15.81:0/90656277 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1c628d65-bba1-4a42-bc80-dc5197e22e2b", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3764451965 0 1371665406) 0x7f62c0146460 con 0x7f62c013d120 2024-03-25T18:12:42.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:42.152+0000 7f62bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1c628d65-bba1-4a42-bc80-dc5197e22e2b", "id": 7}] 2024-03-25T18:12:42.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:42.152+0000 7f62bcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:42.154 INFO:teuthology.orchestra.run.smithi081.stdout:7 2024-03-25T18:12:42.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:42.153+0000 7f62c6932640 10 librados: watch_flush enter 2024-03-25T18:12:42.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:42.153+0000 7f62c6932640 10 librados: watch_flush exit 2024-03-25T18:12:42.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:42.153+0000 7f62c6932640 20 client.4161.objecter shutdown clearing up homeless session... 2024-03-25T18:12:42.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:42.153+0000 7f62c6932640 10 client.4161.objecter successfully canceled tick 2024-03-25T18:12:42.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:42.153+0000 7f62c6932640 10 monclient: shutdown 2024-03-25T18:12:42.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:42.153+0000 7f62c6932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:42.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:42.153+0000 7f62c6932640 1 -- 172.21.15.81:0/90656277 >> v1:172.21.15.63:6789/0 conn(0x7f62c013d120 legacy=0x7f62c01e2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:42.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:42.154+0000 7f62c6932640 1 -- 172.21.15.81:0/90656277 shutdown_connections 2024-03-25T18:12:42.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:42.154+0000 7f62c6932640 1 -- 172.21.15.81:0/90656277 >> 172.21.15.81:0/90656277 conn(0x7f62c00b4bb0 msgr2=0x7f62c00b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:42.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:42.154+0000 7f62c6932640 1 -- 172.21.15.81:0/90656277 shutdown_connections 2024-03-25T18:12:42.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:42.154+0000 7f62c6932640 1 -- 172.21.15.81:0/90656277 wait complete. 2024-03-25T18:12:42.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T18:12:42.154+0000 7f62c6932640 1 librados: shutdown 2024-03-25T18:12:42.164 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:12:42.165 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:12:42.165 DEBUG:teuthology.orchestra.run.smithi063:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:12:42.167 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:12:42.167 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:12:42.167 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:12:42.167 DEBUG:teuthology.orchestra.run.smithi063:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:12:42.169 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:12:42.169 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:12:42.169 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:12:42.169 DEBUG:teuthology.orchestra.run.smithi063:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:12:42.172 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:12:42.172 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:12:42.172 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:12:42.172 DEBUG:teuthology.orchestra.run.smithi063:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:12:42.174 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:12:42.174 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:12:42.174 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:12:42.174 DEBUG:teuthology.orchestra.run.smithi081:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:12:42.176 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:12:42.177 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:12:42.177 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:12:42.177 DEBUG:teuthology.orchestra.run.smithi081:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:12:42.179 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:12:42.179 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:12:42.179 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:12:42.179 DEBUG:teuthology.orchestra.run.smithi081:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:12:42.181 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:12:42.181 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:12:42.181 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:12:42.181 DEBUG:teuthology.orchestra.run.smithi081:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:12:42.183 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:12:42.183 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:42.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.339+0000 7f2516132640 10 monclient: get_monmap_and_config 2024-03-25T18:12:42.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.339+0000 7f2516132640 10 monclient: build_initial_monmap 2024-03-25T18:12:42.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.339+0000 7f2516132640 10 monclient: monmap: 2024-03-25T18:12:42.341 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-03-25T18:12:42.341 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:42.341 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T18:12:42.340757+0000 2024-03-25T18:12:42.341 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T18:12:42.340757+0000 2024-03-25T18:12:42.341 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:42.341 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:42.341 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:42.341 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:12:42.341 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:12:42.341 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T18:12:42.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.340+0000 7f2516132640 1 Processor -- start 2024-03-25T18:12:42.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.340+0000 7f2516132640 1 -- start start 2024-03-25T18:12:42.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.340+0000 7f2516132640 10 monclient: init 2024-03-25T18:12:42.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.341+0000 7f2516132640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:42.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.341+0000 7f2516132640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:42.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.341+0000 7f2516132640 10 monclient(hunting): picked mon.noname-c con 0x7f25100b4700 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:42.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.341+0000 7f2516132640 10 monclient(hunting): picked mon.noname-a con 0x7f2510142a20 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:42.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.341+0000 7f2516132640 10 monclient(hunting): picked mon.noname-b con 0x7f25100a96b0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:42.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.341+0000 7f2516132640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25100a9d60 con 0x7f2510142a20 2024-03-25T18:12:42.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.341+0000 7f2516132640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f251013ff10 con 0x7f25100b4700 2024-03-25T18:12:42.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.341+0000 7f2516132640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25100aa1f0 con 0x7f25100a96b0 2024-03-25T18:12:42.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.341+0000 7f2516132640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:42.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.341+0000 7f2516132640 10 monclient(hunting): authenticate will time out at 7456.528030s 2024-03-25T18:12:42.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.345+0000 7f250ffff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f25100a96b0 0x7f251013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:42096/0 (socket says 172.21.15.63:42096) 2024-03-25T18:12:42.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.345+0000 7f250ffff640 1 -- 172.21.15.63:0/2458605027 learned_addr learned my addr 172.21.15.63:0/2458605027 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:12:42.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.557+0000 7f250e7fc640 1 -- 172.21.15.63:0/2458605027 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455563387 0 0) 0x7f251013ff10 con 0x7f25100b4700 2024-03-25T18:12:42.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.557+0000 7f250e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:42.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.557+0000 7f250e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:42.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.557+0000 7f250e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:42.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.557+0000 7f250e7fc640 10 monclient(hunting): my global_id is 4171 2024-03-25T18:12:42.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.557+0000 7f250e7fc640 1 -- 172.21.15.63:0/2458605027 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f8003520 con 0x7f25100b4700 2024-03-25T18:12:42.584 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:42.583+0000 7f0bae267640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:12:42.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.603+0000 7f250e7fc640 1 -- 172.21.15.63:0/2458605027 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572028415 0 0) 0x7f25100a9d60 con 0x7f2510142a20 2024-03-25T18:12:42.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.603+0000 7f250e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:42.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.603+0000 7f250e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:42.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.603+0000 7f250e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:42.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.603+0000 7f250e7fc640 10 monclient(hunting): my global_id is 4164 2024-03-25T18:12:42.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.603+0000 7f250e7fc640 1 -- 172.21.15.63:0/2458605027 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f251013ff10 con 0x7f2510142a20 2024-03-25T18:12:42.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.603+0000 7f250e7fc640 1 -- 172.21.15.63:0/2458605027 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884775920 0 0) 0x7f25100aa1f0 con 0x7f25100a96b0 2024-03-25T18:12:42.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.603+0000 7f250e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:42.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.603+0000 7f250e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:42.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.603+0000 7f250e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:42.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.603+0000 7f250e7fc640 10 monclient(hunting): my global_id is 4172 2024-03-25T18:12:42.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.603+0000 7f250e7fc640 1 -- 172.21.15.63:0/2458605027 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25100a9d60 con 0x7f25100a96b0 2024-03-25T18:12:42.609 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-25T18:12:42.608+0000 7f0bae267640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:12:42.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f250e7fc640 1 -- 172.21.15.63:0/2458605027 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 535071681 0 0) 0x7f25100a9d60 con 0x7f25100a96b0 2024-03-25T18:12:42.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f250e7fc640 1 -- 172.21.15.63:0/2458605027 >> v1:172.21.15.81:6789/0 conn(0x7f25100b4700 legacy=0x7f2510142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:42.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f250e7fc640 1 -- 172.21.15.63:0/2458605027 >> v1:172.21.15.63:6789/0 conn(0x7f2510142a20 legacy=0x7f251013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:42.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f250e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f250e7fc640 1 monclient: found mon.noname-b 2024-03-25T18:12:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f250e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f250e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:12:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f250e7fc640 1 -- 172.21.15.63:0/2458605027 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25100b4ae0 con 0x7f25100a96b0 2024-03-25T18:12:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f250e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f250e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f250e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f2516132640 5 monclient: authenticate success, global_id 4172 2024-03-25T18:12:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f2516132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f250e7fc640 1 -- 172.21.15.63:0/2458605027 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2508001870 con 0x7f25100a96b0 2024-03-25T18:12:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f250e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f250e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f250e7fc640 10 monclient: dump: 2024-03-25T18:12:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T18:12:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:42.611 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:42.611 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:42.611 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:42.611 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:42.611 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:42.611 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:42.611 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:42.611 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T18:12:42.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.608+0000 7f2516132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:42.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.611+0000 7f250e7fc640 1 -- 172.21.15.63:0/2458605027 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2508001b40 con 0x7f25100a96b0 2024-03-25T18:12:42.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.611+0000 7f250e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:42.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.611+0000 7f2516132640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:42.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.611+0000 7f2516132640 4 set_mon_vals no callback set 2024-03-25T18:12:42.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.611+0000 7f250e7fc640 1 -- 172.21.15.63:0/2458605027 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2508001ed0 con 0x7f25100a96b0 2024-03-25T18:12:42.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.611+0000 7f250e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:42.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.611+0000 7f250e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:42.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.611+0000 7f250e7fc640 10 monclient: dump: 2024-03-25T18:12:42.612 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T18:12:42.612 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:42.612 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:42.612 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:42.612 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:42.613 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:42.613 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:42.613 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:42.613 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:42.613 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T18:12:42.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.611+0000 7f2516132640 10 monclient: shutdown 2024-03-25T18:12:42.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.611+0000 7f2516132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:42.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.611+0000 7f2516132640 1 -- 172.21.15.63:0/2458605027 >> v1:172.21.15.63:6790/0 conn(0x7f25100a96b0 legacy=0x7f251013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:42.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.612+0000 7f2516132640 1 -- 172.21.15.63:0/2458605027 shutdown_connections 2024-03-25T18:12:42.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.612+0000 7f2516132640 1 -- 172.21.15.63:0/2458605027 >> 172.21.15.63:0/2458605027 conn(0x7f25100668e0 msgr2=0x7f2510066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:42.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.612+0000 7f2516132640 1 -- 172.21.15.63:0/2458605027 shutdown_connections 2024-03-25T18:12:42.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.612+0000 7f2516132640 1 -- 172.21.15.63:0/2458605027 wait complete. 2024-03-25T18:12:42.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.612+0000 7f2516132640 10 monclient: build_initial_monmap 2024-03-25T18:12:42.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.612+0000 7f2516132640 10 monclient: monmap: 2024-03-25T18:12:42.615 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-03-25T18:12:42.615 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:42.615 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-03-25T18:12:42.615 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-03-25T18:12:42.615 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:42.615 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-03-25T18:12:42.615 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:42.615 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:12:42.615 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:12:42.615 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T18:12:42.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.613+0000 7f2516132640 1 librados: starting msgr at 2024-03-25T18:12:42.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.613+0000 7f2516132640 1 librados: starting objecter 2024-03-25T18:12:42.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.613+0000 7f2516132640 1 Processor -- start 2024-03-25T18:12:42.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.613+0000 7f2516132640 1 -- start start 2024-03-25T18:12:42.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.613+0000 7f2516132640 1 librados: setting wanted keys 2024-03-25T18:12:42.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.613+0000 7f2516132640 1 librados: calling monclient init 2024-03-25T18:12:42.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.613+0000 7f2516132640 10 monclient: init 2024-03-25T18:12:42.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.614+0000 7f2516132640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:42.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.614+0000 7f2516132640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:42.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.614+0000 7f2516132640 10 monclient(hunting): picked mon.noname-b con 0x7f25100a96b0 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:42.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.614+0000 7f2516132640 10 monclient(hunting): picked mon.noname-a con 0x7f25100b4700 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:42.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.615+0000 7f2516132640 10 monclient(hunting): picked mon.noname-c con 0x7f2510142a20 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:42.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.615+0000 7f2516132640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25101d7700 con 0x7f25100b4700 2024-03-25T18:12:42.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.615+0000 7f2516132640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25100ade20 con 0x7f25100a96b0 2024-03-25T18:12:42.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.615+0000 7f2516132640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25101d82a0 con 0x7f2510142a20 2024-03-25T18:12:42.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.615+0000 7f2516132640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:42.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.615+0000 7f2516132640 10 monclient(hunting): authenticate will time out at 7456.801851s 2024-03-25T18:12:42.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.615+0000 7f250ffff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f2510142a20 0x7f25100ac480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:42100/0 (socket says 172.21.15.63:42100) 2024-03-25T18:12:42.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.615+0000 7f250ffff640 1 -- 172.21.15.63:0/1142076449 learned_addr learned my addr 172.21.15.63:0/1142076449 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:12:42.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.620+0000 7f24f7fff640 10 client.?.objecter ms_handle_connect 0x7f2510142a20 2024-03-25T18:12:42.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.620+0000 7f24f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:42.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.623+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955262659 0 0) 0x7f25101d82a0 con 0x7f2510142a20 2024-03-25T18:12:42.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.623+0000 7f24f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:42.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.623+0000 7f24f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:42.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.623+0000 7f24f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:42.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.623+0000 7f24f7fff640 10 monclient(hunting): my global_id is 4175 2024-03-25T18:12:42.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.623+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24fc004eb0 con 0x7f2510142a20 2024-03-25T18:12:42.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.628+0000 7f24f7fff640 10 client.?.objecter ms_handle_connect 0x7f25100a96b0 2024-03-25T18:12:42.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.628+0000 7f24f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:42.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.632+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208124849 0 0) 0x7f25100ade20 con 0x7f25100a96b0 2024-03-25T18:12:42.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.632+0000 7f24f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:42.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.632+0000 7f24f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:42.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.632+0000 7f24f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:42.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.632+0000 7f24f7fff640 10 monclient(hunting): my global_id is 4174 2024-03-25T18:12:42.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.632+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25101d82a0 con 0x7f25100a96b0 2024-03-25T18:12:42.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.636+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1459425691 0 0) 0x7f24fc004eb0 con 0x7f2510142a20 2024-03-25T18:12:42.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.636+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25100ade20 con 0x7f2510142a20 2024-03-25T18:12:42.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.636+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2508001930 con 0x7f2510142a20 2024-03-25T18:12:42.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.636+0000 7f24f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:42.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.636+0000 7f24f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:42.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.636+0000 7f24f7fff640 10 monclient(hunting): dump: 2024-03-25T18:12:42.637 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T18:12:42.637 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:42.637 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:42.637 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:42.637 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:42.637 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:42.637 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:42.637 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:42.637 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:42.637 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T18:12:42.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.636+0000 7f24f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:42.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.640+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 464786984 0 0) 0x7f25101d82a0 con 0x7f25100a96b0 2024-03-25T18:12:42.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.640+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24fc004eb0 con 0x7f25100a96b0 2024-03-25T18:12:42.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.640+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2504001950 con 0x7f25100a96b0 2024-03-25T18:12:42.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.640+0000 7f24f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:42.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.640+0000 7f24f7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:42.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.640+0000 7f24f7fff640 10 monclient(hunting): dump: 2024-03-25T18:12:42.642 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T18:12:42.642 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:42.642 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:42.642 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:42.642 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:42.642 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:42.642 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:42.642 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:42.642 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:42.643 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T18:12:42.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.640+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3938256981 0 0) 0x7f25100ade20 con 0x7f2510142a20 2024-03-25T18:12:42.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.640+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 >> v1:172.21.15.81:6789/0 conn(0x7f25100a96b0 legacy=0x7f25101d7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:42.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.640+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 >> v1:172.21.15.63:6789/0 conn(0x7f25100b4700 legacy=0x7f25101d46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:42.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.640+0000 7f24f7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:42.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.640+0000 7f24f7fff640 1 monclient: found mon.c 2024-03-25T18:12:42.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.640+0000 7f24f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:42.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.640+0000 7f24f7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:42.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.640+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25101d4df0 con 0x7f2510142a20 2024-03-25T18:12:42.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.641+0000 7f24f7fff640 10 monclient: _finish_auth 0 2024-03-25T18:12:42.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.641+0000 7f24f7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:42.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.641+0000 7f24f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:42.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.641+0000 7f2516132640 5 monclient: authenticate success, global_id 4175 2024-03-25T18:12:42.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.641+0000 7f2516132640 10 monclient: _renew_subs 2024-03-25T18:12:42.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.641+0000 7f2516132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:42.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.641+0000 7f2516132640 1 -- 172.21.15.63:0/1142076449 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25101d5040 con 0x7f2510142a20 2024-03-25T18:12:42.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.641+0000 7f2516132640 10 client.4175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:42.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.641+0000 7f2516132640 10 monclient: _renew_subs 2024-03-25T18:12:42.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.641+0000 7f2516132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:42.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.641+0000 7f2516132640 1 -- 172.21.15.63:0/1142076449 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25101d41b0 con 0x7f2510142a20 2024-03-25T18:12:42.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.641+0000 7f2516132640 1 librados: init done 2024-03-25T18:12:42.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.642+0000 7f2516132640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:42.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.642+0000 7f2516132640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:42.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.642+0000 7f2516132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:42.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.642+0000 7f2516132640 1 -- 172.21.15.63:0/1142076449 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25101d74d0 con 0x7f2510142a20 2024-03-25T18:12:42.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.644+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2508004e10 con 0x7f2510142a20 2024-03-25T18:12:42.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.644+0000 7f24f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:42.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.644+0000 7f250dffb640 4 set_mon_vals no callback set 2024-03-25T18:12:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.646+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2508005000 con 0x7f2510142a20 2024-03-25T18:12:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.646+0000 7f24f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.646+0000 7f24f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.646+0000 7f24f7fff640 10 monclient: dump: 2024-03-25T18:12:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T18:12:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:42.648 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:42.648 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:42.648 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:42.648 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:42.648 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:42.648 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:42.648 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T18:12:42.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.678+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 4113699467 0 0) 0x7f250801afe0 con 0x7f2510142a20 2024-03-25T18:12:42.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.679+0000 7f250f7fe640 10 monclient: get_auth_request con 0x7f24fc06dd60 auth_method 0 2024-03-25T18:12:42.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.679+0000 7f24f7fff640 10 client.4175.objecter ms_handle_connect 0x7f24fc06dd60 2024-03-25T18:12:42.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.929+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f2508054a90 con 0x7f2510142a20 2024-03-25T18:12:42.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.929+0000 7f24f7fff640 10 client.4175.objecter ms_dispatch 0x7f2510000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:42.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.929+0000 7f24f7fff640 3 client.4175.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:42.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.929+0000 7f24f7fff640 3 client.4175.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:42.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.929+0000 7f24f7fff640 20 client.4175.objecter dump_active .. 0 homeless 2024-03-25T18:12:42.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.929+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25080505a0 con 0x7f2510142a20 2024-03-25T18:12:42.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.929+0000 7f24f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:42.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:42.929+0000 7f24f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:43.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.297+0000 7f2516132640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:43.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.297+0000 7f2516132640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:43.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.297+0000 7f2516132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:12:43.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.297+0000 7f2516132640 1 -- 172.21.15.63:0/1142076449 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f251004f730 con 0x7f2510142a20 2024-03-25T18:12:43.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.375+0000 7f24f7fff640 1 -- 172.21.15.63:0/1142076449 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f25080524d0 con 0x7f2510142a20 2024-03-25T18:12:43.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.375+0000 7f24f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:43.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.375+0000 7f24f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:43.376 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T18:12:43.376 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:43.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.377+0000 7f2516132640 10 librados: watch_flush enter 2024-03-25T18:12:43.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.378+0000 7f2516132640 10 librados: watch_flush exit 2024-03-25T18:12:43.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.378+0000 7f2516132640 20 client.4175.objecter shutdown clearing up homeless session... 2024-03-25T18:12:43.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.378+0000 7f2516132640 10 client.4175.objecter successfully canceled tick 2024-03-25T18:12:43.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.378+0000 7f2516132640 1 -- 172.21.15.63:0/1142076449 >> v1:172.21.15.81:6800/3640340873 conn(0x7f24fc06dd60 legacy=0x7f24fc070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.378+0000 7f2516132640 10 monclient: shutdown 2024-03-25T18:12:43.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.378+0000 7f2516132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:43.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.378+0000 7f2516132640 1 -- 172.21.15.63:0/1142076449 >> v1:172.21.15.63:6790/0 conn(0x7f2510142a20 legacy=0x7f25100ac480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.378+0000 7f2516132640 1 -- 172.21.15.63:0/1142076449 shutdown_connections 2024-03-25T18:12:43.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.378+0000 7f2516132640 1 -- 172.21.15.63:0/1142076449 >> 172.21.15.63:0/1142076449 conn(0x7f25100668e0 msgr2=0x7f251013f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:43.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.378+0000 7f2516132640 1 -- 172.21.15.63:0/1142076449 shutdown_connections 2024-03-25T18:12:43.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.378+0000 7f2516132640 1 -- 172.21.15.63:0/1142076449 wait complete. 2024-03-25T18:12:43.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T18:12:43.378+0000 7f2516132640 1 librados: shutdown 2024-03-25T18:12:43.395 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:12:43.395 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:12:44.202 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:44.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.349+0000 7f6c79932640 10 monclient: get_monmap_and_config 2024-03-25T18:12:44.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.349+0000 7f6c79932640 10 monclient: build_initial_monmap 2024-03-25T18:12:44.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.350+0000 7f6c79932640 10 monclient: monmap: 2024-03-25T18:12:44.351 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:12:44.351 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:44.351 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:12:44.351101+0000 2024-03-25T18:12:44.351 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:12:44.351101+0000 2024-03-25T18:12:44.351 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:44.351 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:44.351 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:44.351 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:12:44.351 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:12:44.351 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:44.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.351+0000 7f6c79932640 1 Processor -- start 2024-03-25T18:12:44.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.351+0000 7f6c79932640 1 -- start start 2024-03-25T18:12:44.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.351+0000 7f6c79932640 10 monclient: init 2024-03-25T18:12:44.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.352+0000 7f6c79932640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:44.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.352+0000 7f6c79932640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:44.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.352+0000 7f6c79932640 10 monclient(hunting): picked mon.noname-a con 0x7f6c740b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:44.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.352+0000 7f6c79932640 10 monclient(hunting): picked mon.noname-b con 0x7f6c740b1730 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:44.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.352+0000 7f6c79932640 10 monclient(hunting): picked mon.noname-c con 0x7f6c7414be90 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:44.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.352+0000 7f6c79932640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c740af940 con 0x7f6c740b4d60 2024-03-25T18:12:44.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.352+0000 7f6c79932640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c740b0a70 con 0x7f6c7414be90 2024-03-25T18:12:44.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.352+0000 7f6c79932640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c740b0d50 con 0x7f6c740b1730 2024-03-25T18:12:44.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.352+0000 7f6c79932640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:44.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.352+0000 7f6c79932640 10 monclient(hunting): authenticate will time out at 7458.538541s 2024-03-25T18:12:44.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.352+0000 7f6c6bfff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f6c740b1730 0x7f6c740b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:42112/0 (socket says 172.21.15.63:42112) 2024-03-25T18:12:44.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.352+0000 7f6c6bfff640 1 -- 172.21.15.63:0/3433927079 learned_addr learned my addr 172.21.15.63:0/3433927079 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:12:44.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.356+0000 7f6c727fc640 1 -- 172.21.15.63:0/3433927079 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668788074 0 0) 0x7f6c740b0d50 con 0x7f6c740b1730 2024-03-25T18:12:44.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.356+0000 7f6c727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:44.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.356+0000 7f6c727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:44.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.356+0000 7f6c727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:44.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.356+0000 7f6c727fc640 10 monclient(hunting): my global_id is 4178 2024-03-25T18:12:44.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.356+0000 7f6c727fc640 1 -- 172.21.15.63:0/3433927079 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c60003520 con 0x7f6c740b1730 2024-03-25T18:12:44.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.356+0000 7f6c727fc640 1 -- 172.21.15.63:0/3433927079 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373932646 0 0) 0x7f6c740af940 con 0x7f6c740b4d60 2024-03-25T18:12:44.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.356+0000 7f6c727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:44.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.356+0000 7f6c727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:44.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.356+0000 7f6c727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:44.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.356+0000 7f6c727fc640 10 monclient(hunting): my global_id is 4167 2024-03-25T18:12:44.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.356+0000 7f6c727fc640 1 -- 172.21.15.63:0/3433927079 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c740b0d50 con 0x7f6c740b4d60 2024-03-25T18:12:44.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.357+0000 7f6c727fc640 1 -- 172.21.15.63:0/3433927079 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765361747 0 0) 0x7f6c740b0a70 con 0x7f6c7414be90 2024-03-25T18:12:44.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.357+0000 7f6c727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:44.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.357+0000 7f6c727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:44.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.357+0000 7f6c727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:44.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.357+0000 7f6c727fc640 10 monclient(hunting): my global_id is 4177 2024-03-25T18:12:44.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.357+0000 7f6c727fc640 1 -- 172.21.15.63:0/3433927079 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c740af940 con 0x7f6c7414be90 2024-03-25T18:12:44.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.361+0000 7f6c727fc640 1 -- 172.21.15.63:0/3433927079 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2020813490 0 0) 0x7f6c740b0d50 con 0x7f6c740b4d60 2024-03-25T18:12:44.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.361+0000 7f6c727fc640 1 -- 172.21.15.63:0/3433927079 >> v1:172.21.15.63:6790/0 conn(0x7f6c740b1730 legacy=0x7f6c740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:44.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.361+0000 7f6c727fc640 1 -- 172.21.15.63:0/3433927079 >> v1:172.21.15.81:6789/0 conn(0x7f6c7414be90 legacy=0x7f6c7414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:44.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.361+0000 7f6c727fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:44.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.361+0000 7f6c727fc640 1 monclient: found mon.noname-a 2024-03-25T18:12:44.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.361+0000 7f6c727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:44.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.361+0000 7f6c727fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:12:44.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.361+0000 7f6c727fc640 1 -- 172.21.15.63:0/3433927079 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c740b1030 con 0x7f6c740b4d60 2024-03-25T18:12:44.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.361+0000 7f6c727fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:44.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.361+0000 7f6c727fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:44.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.361+0000 7f6c727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:44.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.361+0000 7f6c79932640 5 monclient: authenticate success, global_id 4167 2024-03-25T18:12:44.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.361+0000 7f6c79932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:44.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.362+0000 7f6c727fc640 1 -- 172.21.15.63:0/3433927079 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6c64001630 con 0x7f6c740b4d60 2024-03-25T18:12:44.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.362+0000 7f6c727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:44.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.362+0000 7f6c727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:44.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.362+0000 7f6c727fc640 10 monclient: dump: 2024-03-25T18:12:44.365 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:44.365 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:44.365 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:44.365 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:44.365 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:44.365 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:44.365 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:44.365 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:44.365 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:44.365 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:44.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.362+0000 7f6c79932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:44.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.364+0000 7f6c727fc640 1 -- 172.21.15.63:0/3433927079 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c64001880 con 0x7f6c740b4d60 2024-03-25T18:12:44.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.364+0000 7f6c727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:44.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.364+0000 7f6c727fc640 1 -- 172.21.15.63:0/3433927079 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6c64001c10 con 0x7f6c740b4d60 2024-03-25T18:12:44.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.364+0000 7f6c727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:44.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.364+0000 7f6c727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:44.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.364+0000 7f6c727fc640 10 monclient: dump: 2024-03-25T18:12:44.368 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:44.368 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:44.368 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:44.369 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:44.369 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:44.369 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:44.369 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:44.369 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:44.369 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:44.369 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:44.369 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.364+0000 7f6c79932640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:44.369 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.364+0000 7f6c79932640 4 set_mon_vals no callback set 2024-03-25T18:12:44.369 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.365+0000 7f6c79932640 10 monclient: shutdown 2024-03-25T18:12:44.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.365+0000 7f6c79932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:44.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.365+0000 7f6c79932640 1 -- 172.21.15.63:0/3433927079 >> v1:172.21.15.63:6789/0 conn(0x7f6c740b4d60 legacy=0x7f6c7413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:44.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.365+0000 7f6c79932640 1 -- 172.21.15.63:0/3433927079 shutdown_connections 2024-03-25T18:12:44.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.365+0000 7f6c79932640 1 -- 172.21.15.63:0/3433927079 >> 172.21.15.63:0/3433927079 conn(0x7f6c740617a0 msgr2=0x7f6c74061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:44.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.365+0000 7f6c79932640 1 -- 172.21.15.63:0/3433927079 shutdown_connections 2024-03-25T18:12:44.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.365+0000 7f6c79932640 1 -- 172.21.15.63:0/3433927079 wait complete. 2024-03-25T18:12:44.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.366+0000 7f6c79932640 10 monclient: build_initial_monmap 2024-03-25T18:12:44.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.366+0000 7f6c79932640 10 monclient: monmap: 2024-03-25T18:12:44.370 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:12:44.370 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:44.370 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:12:44.370 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:12:44.370 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:44.371 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:12:44.371 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:44.371 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:12:44.371 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:12:44.371 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:44.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.367+0000 7f6c79932640 1 librados: starting msgr at 2024-03-25T18:12:44.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.367+0000 7f6c79932640 1 librados: starting objecter 2024-03-25T18:12:44.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.367+0000 7f6c79932640 1 Processor -- start 2024-03-25T18:12:44.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.367+0000 7f6c79932640 1 -- start start 2024-03-25T18:12:44.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.367+0000 7f6c79932640 1 librados: setting wanted keys 2024-03-25T18:12:44.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.367+0000 7f6c79932640 1 librados: calling monclient init 2024-03-25T18:12:44.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.367+0000 7f6c79932640 10 monclient: init 2024-03-25T18:12:44.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.368+0000 7f6c79932640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:44.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.368+0000 7f6c79932640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:44.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.368+0000 7f6c79932640 10 monclient(hunting): picked mon.noname-b con 0x7f6c740b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:44.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.368+0000 7f6c79932640 10 monclient(hunting): picked mon.noname-c con 0x7f6c740b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:44.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.368+0000 7f6c79932640 10 monclient(hunting): picked mon.noname-a con 0x7f6c7414be90 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:44.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.368+0000 7f6c79932640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c74140e50 con 0x7f6c7414be90 2024-03-25T18:12:44.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.368+0000 7f6c79932640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c74138620 con 0x7f6c740b1730 2024-03-25T18:12:44.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.368+0000 7f6c79932640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c74137640 con 0x7f6c740b4d60 2024-03-25T18:12:44.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.368+0000 7f6c79932640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:44.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.368+0000 7f6c79932640 10 monclient(hunting): authenticate will time out at 7458.555185s 2024-03-25T18:12:44.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.369+0000 7f6c6bfff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f6c740b4d60 0x7f6c74140740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:42122/0 (socket says 172.21.15.63:42122) 2024-03-25T18:12:44.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.369+0000 7f6c6bfff640 1 -- 172.21.15.63:0/737810549 learned_addr learned my addr 172.21.15.63:0/737810549 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:12:44.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.369+0000 7f6c70ff9640 10 client.?.objecter ms_handle_connect 0x7f6c740b4d60 2024-03-25T18:12:44.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.369+0000 7f6c70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:44.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.369+0000 7f6c70ff9640 10 client.?.objecter ms_handle_connect 0x7f6c7414be90 2024-03-25T18:12:44.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.369+0000 7f6c70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:44.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.369+0000 7f6c70ff9640 10 client.?.objecter ms_handle_connect 0x7f6c740b1730 2024-03-25T18:12:44.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.369+0000 7f6c70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:44.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.372+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021122260 0 0) 0x7f6c74140e50 con 0x7f6c7414be90 2024-03-25T18:12:44.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.372+0000 7f6c70ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:44.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.372+0000 7f6c70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:44.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.372+0000 7f6c70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:44.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.372+0000 7f6c70ff9640 10 monclient(hunting): my global_id is 4170 2024-03-25T18:12:44.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.372+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c54004eb0 con 0x7f6c7414be90 2024-03-25T18:12:44.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.372+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188751030 0 0) 0x7f6c74137640 con 0x7f6c740b4d60 2024-03-25T18:12:44.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.373+0000 7f6c70ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:44.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.373+0000 7f6c70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:44.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.373+0000 7f6c70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:44.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.373+0000 7f6c70ff9640 10 monclient(hunting): my global_id is 4181 2024-03-25T18:12:44.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.373+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c74140e50 con 0x7f6c740b4d60 2024-03-25T18:12:44.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.373+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756112392 0 0) 0x7f6c74138620 con 0x7f6c740b1730 2024-03-25T18:12:44.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.373+0000 7f6c70ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:44.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.373+0000 7f6c70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:44.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.373+0000 7f6c70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:44.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.373+0000 7f6c70ff9640 10 monclient(hunting): my global_id is 4180 2024-03-25T18:12:44.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.373+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c74137640 con 0x7f6c740b1730 2024-03-25T18:12:44.382 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.380+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3166910293 0 0) 0x7f6c74137640 con 0x7f6c740b1730 2024-03-25T18:12:44.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.380+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c74138620 con 0x7f6c740b1730 2024-03-25T18:12:44.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.380+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6c640019f0 con 0x7f6c740b1730 2024-03-25T18:12:44.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.380+0000 7f6c70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:44.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.380+0000 7f6c70ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:44.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.380+0000 7f6c70ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:44.383 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:44.383 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:44.383 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:44.383 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:44.383 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:44.383 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:44.383 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:44.383 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:44.383 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:44.383 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:44.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.380+0000 7f6c70ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:44.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.380+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 264545455 0 0) 0x7f6c74140e50 con 0x7f6c740b4d60 2024-03-25T18:12:44.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.381+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c74137640 con 0x7f6c740b4d60 2024-03-25T18:12:44.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.381+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2418002597 0 0) 0x7f6c54004eb0 con 0x7f6c7414be90 2024-03-25T18:12:44.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.381+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c74140e50 con 0x7f6c7414be90 2024-03-25T18:12:44.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.381+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6c6c001bd0 con 0x7f6c7414be90 2024-03-25T18:12:44.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.381+0000 7f6c70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:44.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.381+0000 7f6c70ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:44.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.381+0000 7f6c70ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:44.384 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:44.384 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:44.384 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:44.384 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:44.384 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:44.384 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:44.385 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:44.385 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:44.385 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:44.385 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:44.385 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.381+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6c5c001970 con 0x7f6c740b4d60 2024-03-25T18:12:44.385 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.381+0000 7f6c70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:44.385 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.381+0000 7f6c70ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:44.385 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.381+0000 7f6c70ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:44.385 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:44.385 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:44.385 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:44.385 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:44.385 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:44.385 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:44.385 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:44.386 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:44.386 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:44.386 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:44.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.386+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1356210738 0 0) 0x7f6c74138620 con 0x7f6c740b1730 2024-03-25T18:12:44.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.386+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 >> v1:172.21.15.63:6790/0 conn(0x7f6c740b4d60 legacy=0x7f6c74140740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:44.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.386+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 >> v1:172.21.15.63:6789/0 conn(0x7f6c7414be90 legacy=0x7f6c741e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:44.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.386+0000 7f6c70ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:12:44.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.386+0000 7f6c70ff9640 1 monclient: found mon.b 2024-03-25T18:12:44.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.386+0000 7f6c70ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:44.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.386+0000 7f6c70ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:44.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.386+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c74135b00 con 0x7f6c740b1730 2024-03-25T18:12:44.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.387+0000 7f6c70ff9640 10 monclient: _finish_auth 0 2024-03-25T18:12:44.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.387+0000 7f6c70ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:12:44.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.387+0000 7f6c70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:44.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.387+0000 7f6c79932640 5 monclient: authenticate success, global_id 4180 2024-03-25T18:12:44.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.387+0000 7f6c79932640 10 monclient: _renew_subs 2024-03-25T18:12:44.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.387+0000 7f6c79932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:44.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.387+0000 7f6c79932640 1 -- 172.21.15.63:0/737810549 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c741362a0 con 0x7f6c740b1730 2024-03-25T18:12:44.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.387+0000 7f6c79932640 10 client.4180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:44.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.387+0000 7f6c79932640 10 monclient: _renew_subs 2024-03-25T18:12:44.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.387+0000 7f6c79932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:44.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.387+0000 7f6c79932640 1 -- 172.21.15.63:0/737810549 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c74144ca0 con 0x7f6c740b1730 2024-03-25T18:12:44.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.387+0000 7f6c79932640 1 librados: init done 2024-03-25T18:12:44.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.389+0000 7f6c79932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:44.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.389+0000 7f6c79932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:44.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.389+0000 7f6c79932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:44.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.389+0000 7f6c79932640 1 -- 172.21.15.63:0/737810549 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c74144fa0 con 0x7f6c740b1730 2024-03-25T18:12:44.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.607+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c64001b50 con 0x7f6c740b1730 2024-03-25T18:12:44.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.607+0000 7f6c70ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:44.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.607+0000 7f6c71ffb640 4 set_mon_vals no callback set 2024-03-25T18:12:44.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.826+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6c640038a0 con 0x7f6c740b1730 2024-03-25T18:12:44.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.826+0000 7f6c70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:44.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.826+0000 7f6c70ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:44.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.826+0000 7f6c70ff9640 10 monclient: dump: 2024-03-25T18:12:44.827 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:44.828 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:44.828 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:44.828 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:44.828 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:44.828 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:44.828 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:44.828 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:44.828 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:44.828 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:44.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.827+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f6c64019870 con 0x7f6c740b1730 2024-03-25T18:12:44.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.828+0000 7f6c6bfff640 10 monclient: get_auth_request con 0x7f6c5406e1f0 auth_method 0 2024-03-25T18:12:44.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.828+0000 7f6c70ff9640 10 client.4180.objecter ms_handle_connect 0x7f6c5406e1f0 2024-03-25T18:12:44.834 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.834+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f6c64053480 con 0x7f6c740b1730 2024-03-25T18:12:44.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.834+0000 7f6c70ff9640 10 client.4180.objecter ms_dispatch 0x7f6c74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:44.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.834+0000 7f6c70ff9640 3 client.4180.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:44.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.834+0000 7f6c70ff9640 3 client.4180.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:44.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.834+0000 7f6c70ff9640 20 client.4180.objecter dump_active .. 0 homeless 2024-03-25T18:12:44.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.837+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6c6404edc0 con 0x7f6c740b1730 2024-03-25T18:12:44.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.838+0000 7f6c70ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:44.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:44.838+0000 7f6c70ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:45.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.204+0000 7f6c79932640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:45.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.204+0000 7f6c79932640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:45.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.204+0000 7f6c79932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:45.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.204+0000 7f6c79932640 1 -- 172.21.15.63:0/737810549 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c741377d0 con 0x7f6c740b1730 2024-03-25T18:12:45.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.276+0000 7f6c70ff9640 1 -- 172.21.15.63:0/737810549 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f6c64050ec0 con 0x7f6c740b1730 2024-03-25T18:12:45.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.276+0000 7f6c70ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:45.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.276+0000 7f6c70ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:45.278 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:12:45.278 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:45.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.279+0000 7f6c79932640 10 librados: watch_flush enter 2024-03-25T18:12:45.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.279+0000 7f6c79932640 10 librados: watch_flush exit 2024-03-25T18:12:45.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.279+0000 7f6c79932640 20 client.4180.objecter shutdown clearing up homeless session... 2024-03-25T18:12:45.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.279+0000 7f6c79932640 10 client.4180.objecter successfully canceled tick 2024-03-25T18:12:45.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.279+0000 7f6c79932640 1 -- 172.21.15.63:0/737810549 >> v1:172.21.15.81:6800/3640340873 conn(0x7f6c5406e1f0 legacy=0x7f6c540706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:45.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.279+0000 7f6c79932640 10 monclient: shutdown 2024-03-25T18:12:45.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.279+0000 7f6c79932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:45.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.279+0000 7f6c79932640 1 -- 172.21.15.63:0/737810549 >> v1:172.21.15.81:6789/0 conn(0x7f6c740b1730 legacy=0x7f6c74136e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.279+0000 7f6c79932640 1 -- 172.21.15.63:0/737810549 shutdown_connections 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.279+0000 7f6c79932640 1 -- 172.21.15.63:0/737810549 >> 172.21.15.63:0/737810549 conn(0x7f6c740617a0 msgr2=0x7f6c740b6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.280+0000 7f6c79932640 1 -- 172.21.15.63:0/737810549 shutdown_connections 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.280+0000 7f6c79932640 1 -- 172.21.15.63:0/737810549 wait complete. 2024-03-25T18:12:45.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:45.280+0000 7f6c79932640 1 librados: shutdown 2024-03-25T18:12:45.294 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:48.987 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:06.742 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:48.993 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:06.753 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:49.004 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:06.765 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:49.048 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:06.804 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:49.063 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:06.823 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:49.075 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:06.836 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:49.302 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:07.059 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:49.365 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:07.122 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:49.414 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:07.154 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:49.475 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:07.225 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:49.484 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:07.224 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:49.552 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:07.302 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:49.612 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-03-25T18:12:49.602+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:12:49.614 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-03-25T18:12:49.608+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:12:49.629 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-03-25T18:12:49.623+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:12:49.784 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:07.527 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:49.818 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:07.576 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:49.847 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:07.590 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:49.878 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:07.636 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:49.953 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-03-25T18:12:49.946+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:12:50.031 INFO:tasks.ceph.osd.3.smithi063.stderr:2024-03-25T18:12:50.022+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:12:50.098 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:07.858 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:50.108 INFO:tasks.ceph.osd.0.smithi063.stderr:2024-03-25T18:12:50.099+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:12:50.110 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:07.871 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:50.113 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:07.869 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:50.391 INFO:tasks.ceph.osd.2.smithi063.stderr:2024-03-25T18:12:50.381+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:12:50.435 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:08.191 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:50.469 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:08.228 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:50.475 INFO:tasks.ceph.osd.1.smithi063.stderr:2024-03-25T18:12:50.469+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:12:50.489 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:08.230 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:50.492 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:08.252 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:50.492 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:08.247 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:50.570 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:08.321 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:50.788 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:08.548 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:50.799 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:08.555 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:50.810 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:08.570 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:50.830 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:08.586 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:50.858 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:08.598 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:50.873 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:08.615 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:50.938 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:08.688 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:50.945 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:08.703 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.061 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:08.821 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.087 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:08.847 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.126 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:08.881 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.165 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:08.922 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.176 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:08.917 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.240 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:08.983 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.303 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:09.053 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.315 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:09.073 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.328 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:09.088 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.352 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:09.112 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.415 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:09.170 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.442 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:09.182 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.456 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:09.213 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.592 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:09.352 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.597 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:09.340 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.607 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:09.358 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.624 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:09.384 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.645 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:09.403 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.706 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:09.446 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.711 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:09.466 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.752 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:09.509 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.868 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:09.628 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.886 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:09.629 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.899 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:09.659 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.905 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:09.656 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.915 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:09.673 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:51.970 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:09.710 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.004 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:09.759 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.051 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:09.807 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.099 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:52.136 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:09.896 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.168 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:09.928 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.188+0000 7f0252874640 10 monclient: get_monmap_and_config 2024-03-25T18:12:52.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.188+0000 7f0252874640 10 monclient: build_initial_monmap 2024-03-25T18:12:52.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.189+0000 7f0252874640 10 monclient: monmap: 2024-03-25T18:12:52.190 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:12:52.190 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:52.190 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:12:52.190115+0000 2024-03-25T18:12:52.190 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:12:52.190115+0000 2024-03-25T18:12:52.190 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:52.190 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:52.190 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:52.190 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:12:52.190 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:12:52.190 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:52.191 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:09.948 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.191 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.189+0000 7f0252874640 1 Processor -- start 2024-03-25T18:12:52.191 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.189+0000 7f0252874640 1 -- start start 2024-03-25T18:12:52.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.189+0000 7f0252874640 10 monclient: init 2024-03-25T18:12:52.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.190+0000 7f0252874640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:52.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.190+0000 7f0252874640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:52.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.190+0000 7f0252874640 10 monclient(hunting): picked mon.noname-b con 0x7f024c0a96e0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:52.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.190+0000 7f0252874640 10 monclient(hunting): picked mon.noname-c con 0x7f024c0b1b10 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:52.192 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:09.933 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.190+0000 7f0252874640 10 monclient(hunting): picked mon.noname-a con 0x7f024c146060 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:52.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.190+0000 7f0252874640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c140040 con 0x7f024c146060 2024-03-25T18:12:52.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.190+0000 7f0252874640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c0b1fa0 con 0x7f024c0b1b10 2024-03-25T18:12:52.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.190+0000 7f0252874640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c0b24c0 con 0x7f024c0a96e0 2024-03-25T18:12:52.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.190+0000 7f0252874640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:52.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.190+0000 7f0252874640 10 monclient(hunting): authenticate will time out at 7466.376719s 2024-03-25T18:12:52.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.190+0000 7f024bfff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f024c0a96e0 0x7f024c0b0ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:37206/0 (socket says 172.21.15.63:37206) 2024-03-25T18:12:52.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.190+0000 7f024bfff640 1 -- 172.21.15.63:0/3122229037 learned_addr learned my addr 172.21.15.63:0/3122229037 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:12:52.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.192+0000 7f024affd640 1 -- 172.21.15.63:0/3122229037 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913438485 0 0) 0x7f024c0b24c0 con 0x7f024c0a96e0 2024-03-25T18:12:52.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.192+0000 7f024affd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:52.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.192+0000 7f024affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:52.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.192+0000 7f024affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:52.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.192+0000 7f024affd640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:12:52.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.192+0000 7f024affd640 1 -- 172.21.15.63:0/3122229037 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0234003520 con 0x7f024c0a96e0 2024-03-25T18:12:52.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.193+0000 7f024affd640 1 -- 172.21.15.63:0/3122229037 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638532672 0 0) 0x7f024c0b1fa0 con 0x7f024c0b1b10 2024-03-25T18:12:52.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.193+0000 7f024affd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:52.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.193+0000 7f024affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:52.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.193+0000 7f024affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:52.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.193+0000 7f024affd640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:12:52.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.193+0000 7f024affd640 1 -- 172.21.15.63:0/3122229037 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f024c0b24c0 con 0x7f024c0b1b10 2024-03-25T18:12:52.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.193+0000 7f024affd640 1 -- 172.21.15.63:0/3122229037 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579414708 0 0) 0x7f024c140040 con 0x7f024c146060 2024-03-25T18:12:52.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.193+0000 7f024affd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:52.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.193+0000 7f024affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:52.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.193+0000 7f024affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:52.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.193+0000 7f024affd640 10 monclient(hunting): my global_id is 4197 2024-03-25T18:12:52.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.193+0000 7f024affd640 1 -- 172.21.15.63:0/3122229037 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f024c0b1fa0 con 0x7f024c146060 2024-03-25T18:12:52.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f024affd640 1 -- 172.21.15.63:0/3122229037 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 223004497 0 0) 0x7f024c0b24c0 con 0x7f024c0b1b10 2024-03-25T18:12:52.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f024affd640 1 -- 172.21.15.63:0/3122229037 >> v1:172.21.15.63:6790/0 conn(0x7f024c0a96e0 legacy=0x7f024c0b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:52.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f024affd640 1 -- 172.21.15.63:0/3122229037 >> v1:172.21.15.63:6789/0 conn(0x7f024c146060 legacy=0x7f024c14dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:52.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f024affd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:52.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f024affd640 1 monclient: found mon.noname-c 2024-03-25T18:12:52.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f024affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:52.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f024affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:12:52.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f024affd640 1 -- 172.21.15.63:0/3122229037 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c0b15b0 con 0x7f024c0b1b10 2024-03-25T18:12:52.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f024affd640 10 monclient: _finish_auth 0 2024-03-25T18:12:52.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f024affd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:52.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f024affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:52.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f024affd640 1 -- 172.21.15.63:0/3122229037 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f02400016c0 con 0x7f024c0b1b10 2024-03-25T18:12:52.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f0252874640 5 monclient: authenticate success, global_id 4207 2024-03-25T18:12:52.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f0252874640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:52.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f024affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:52.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f024affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:52.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f024affd640 10 monclient: dump: 2024-03-25T18:12:52.198 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:52.198 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:52.199 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:52.199 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:52.199 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:52.199 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:52.199 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:52.199 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:52.199 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:52.199 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:52.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.196+0000 7f0252874640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:52.199 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:09.949 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.198+0000 7f024affd640 1 -- 172.21.15.63:0/3122229037 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0240001820 con 0x7f024c0b1b10 2024-03-25T18:12:52.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.198+0000 7f024affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:52.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.198+0000 7f024affd640 1 -- 172.21.15.63:0/3122229037 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f0240001c30 con 0x7f024c0b1b10 2024-03-25T18:12:52.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.198+0000 7f0252874640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:52.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.198+0000 7f0252874640 4 set_mon_vals no callback set 2024-03-25T18:12:52.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.198+0000 7f024affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:52.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.198+0000 7f024affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:52.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.198+0000 7f024affd640 10 monclient: dump: 2024-03-25T18:12:52.202 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:52.202 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:52.202 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:52.202 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:52.202 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:52.203 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:52.203 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:52.203 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:52.203 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:52.203 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:52.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.199+0000 7f0252874640 10 monclient: shutdown 2024-03-25T18:12:52.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.199+0000 7f0252874640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:52.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.199+0000 7f0252874640 1 -- 172.21.15.63:0/3122229037 >> v1:172.21.15.81:6789/0 conn(0x7f024c0b1b10 legacy=0x7f024c145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:52.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.199+0000 7f0252874640 1 -- 172.21.15.63:0/3122229037 shutdown_connections 2024-03-25T18:12:52.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.199+0000 7f0252874640 1 -- 172.21.15.63:0/3122229037 >> 172.21.15.63:0/3122229037 conn(0x7f024c0b7ea0 msgr2=0x7f024c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:52.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.199+0000 7f0252874640 1 -- 172.21.15.63:0/3122229037 shutdown_connections 2024-03-25T18:12:52.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.199+0000 7f0252874640 1 -- 172.21.15.63:0/3122229037 wait complete. 2024-03-25T18:12:52.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.199+0000 7f0252874640 10 monclient: build_initial_monmap 2024-03-25T18:12:52.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.199+0000 7f0252874640 10 monclient: monmap: 2024-03-25T18:12:52.204 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:12:52.204 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:52.204 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:12:52.204 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:12:52.204 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:52.204 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:12:52.204 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:52.204 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:12:52.204 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:12:52.204 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:52.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.200+0000 7f0252874640 1 librados: starting msgr at 2024-03-25T18:12:52.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.200+0000 7f0252874640 1 librados: starting objecter 2024-03-25T18:12:52.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.200+0000 7f0252874640 1 Processor -- start 2024-03-25T18:12:52.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.200+0000 7f0252874640 1 -- start start 2024-03-25T18:12:52.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.200+0000 7f0252874640 1 librados: setting wanted keys 2024-03-25T18:12:52.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.200+0000 7f0252874640 1 librados: calling monclient init 2024-03-25T18:12:52.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.200+0000 7f0252874640 10 monclient: init 2024-03-25T18:12:52.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.201+0000 7f0252874640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:52.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.201+0000 7f0252874640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:52.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.201+0000 7f0252874640 10 monclient(hunting): picked mon.noname-b con 0x7f024c0a96e0 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:52.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.201+0000 7f0252874640 10 monclient(hunting): picked mon.noname-c con 0x7f024c146060 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:52.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.201+0000 7f0252874640 10 monclient(hunting): picked mon.noname-a con 0x7f024c0af970 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:52.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.201+0000 7f0252874640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c0af7c0 con 0x7f024c0af970 2024-03-25T18:12:52.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.201+0000 7f0252874640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c0afde0 con 0x7f024c0a96e0 2024-03-25T18:12:52.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.201+0000 7f0252874640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c1201e0 con 0x7f024c146060 2024-03-25T18:12:52.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.201+0000 7f0252874640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:52.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.201+0000 7f0252874640 10 monclient(hunting): authenticate will time out at 7466.387391s 2024-03-25T18:12:52.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.201+0000 7f024b7fe640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f024c146060 0x7f024c1d7120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:37220/0 (socket says 172.21.15.63:37220) 2024-03-25T18:12:52.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.201+0000 7f024b7fe640 1 -- 172.21.15.63:0/2332750669 learned_addr learned my addr 172.21.15.63:0/2332750669 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:12:52.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.201+0000 7f0248ff9640 10 client.?.objecter ms_handle_connect 0x7f024c146060 2024-03-25T18:12:52.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.201+0000 7f0248ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:52.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.201+0000 7f0248ff9640 10 client.?.objecter ms_handle_connect 0x7f024c0a96e0 2024-03-25T18:12:52.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.201+0000 7f0248ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:52.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.202+0000 7f0248ff9640 10 client.?.objecter ms_handle_connect 0x7f024c0af970 2024-03-25T18:12:52.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.202+0000 7f0248ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:52.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.203+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932346567 0 0) 0x7f024c0afde0 con 0x7f024c0a96e0 2024-03-25T18:12:52.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.203+0000 7f0248ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:52.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.203+0000 7f0248ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:52.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.203+0000 7f0248ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:52.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.203+0000 7f0248ff9640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:12:52.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.203+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0238004eb0 con 0x7f024c0a96e0 2024-03-25T18:12:52.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.204+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423029102 0 0) 0x7f024c0af7c0 con 0x7f024c0af970 2024-03-25T18:12:52.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.204+0000 7f0248ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:52.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.204+0000 7f0248ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:52.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.204+0000 7f0248ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:52.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.204+0000 7f0248ff9640 10 monclient(hunting): my global_id is 4200 2024-03-25T18:12:52.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.204+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f024c0afde0 con 0x7f024c0af970 2024-03-25T18:12:52.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.204+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688113679 0 0) 0x7f024c1201e0 con 0x7f024c146060 2024-03-25T18:12:52.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.204+0000 7f0248ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:52.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.204+0000 7f0248ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:52.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.204+0000 7f0248ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:52.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.204+0000 7f0248ff9640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:12:52.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.204+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f024c0af7c0 con 0x7f024c146060 2024-03-25T18:12:52.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.208+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2789260033 0 0) 0x7f0238004eb0 con 0x7f024c0a96e0 2024-03-25T18:12:52.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.208+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f024c1201e0 con 0x7f024c0a96e0 2024-03-25T18:12:52.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.208+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f023c001950 con 0x7f024c0a96e0 2024-03-25T18:12:52.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.208+0000 7f0248ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:52.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.208+0000 7f0248ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:52.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.208+0000 7f0248ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:52.209 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:52.209 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:52.210 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:52.210 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:52.210 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:52.210 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:52.210 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:52.210 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:52.210 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:52.210 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:52.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.208+0000 7f0248ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:52.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.209+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1210737829 0 0) 0x7f024c0af7c0 con 0x7f024c146060 2024-03-25T18:12:52.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.209+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0238004eb0 con 0x7f024c146060 2024-03-25T18:12:52.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.209+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f0240001680 con 0x7f024c146060 2024-03-25T18:12:52.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.209+0000 7f0248ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:52.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.209+0000 7f0248ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:52.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.209+0000 7f0248ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:52.211 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:52.212 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:52.212 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:52.212 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:52.212 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:52.212 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:52.212 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:52.212 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:52.212 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:52.212 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:52.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.209+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3279845138 0 0) 0x7f024c0afde0 con 0x7f024c0af970 2024-03-25T18:12:52.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.209+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f024c0af7c0 con 0x7f024c0af970 2024-03-25T18:12:52.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.209+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f0244001b50 con 0x7f024c0af970 2024-03-25T18:12:52.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.209+0000 7f0248ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:52.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.209+0000 7f0248ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:52.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.209+0000 7f0248ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:52.213 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:52.213 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:52.213 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:52.213 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:52.213 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:52.213 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:52.213 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:52.213 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:52.213 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:52.213 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:52.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1436587511 0 0) 0x7f024c1201e0 con 0x7f024c0a96e0 2024-03-25T18:12:52.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 >> v1:172.21.15.63:6790/0 conn(0x7f024c146060 legacy=0x7f024c1d7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:52.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 >> v1:172.21.15.63:6789/0 conn(0x7f024c0af970 legacy=0x7f024c1d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:52.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0248ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:12:52.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0248ff9640 1 monclient: found mon.b 2024-03-25T18:12:52.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0248ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:52.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0248ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:52.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c0b0040 con 0x7f024c0a96e0 2024-03-25T18:12:52.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0248ff9640 10 monclient: _finish_auth 0 2024-03-25T18:12:52.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0248ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:12:52.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0248ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:52.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0252874640 5 monclient: authenticate success, global_id 4210 2024-03-25T18:12:52.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0252874640 10 monclient: _renew_subs 2024-03-25T18:12:52.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0252874640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:52.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0252874640 1 -- 172.21.15.63:0/2332750669 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f024c1e2c50 con 0x7f024c0a96e0 2024-03-25T18:12:52.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0252874640 10 client.4210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:52.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0252874640 10 monclient: _renew_subs 2024-03-25T18:12:52.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0252874640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:52.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0252874640 1 -- 172.21.15.63:0/2332750669 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f024c1e3120 con 0x7f024c0a96e0 2024-03-25T18:12:52.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.211+0000 7f0252874640 1 librados: init done 2024-03-25T18:12:52.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.212+0000 7f0252874640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:52.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.212+0000 7f0252874640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:52.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.212+0000 7f0252874640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:52.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.212+0000 7f0252874640 1 -- 172.21.15.63:0/2332750669 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f024c0a9ac0 con 0x7f024c0a96e0 2024-03-25T18:12:52.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.225+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f023c001d60 con 0x7f024c0a96e0 2024-03-25T18:12:52.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.225+0000 7f0248ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:52.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.225+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f023c004980 con 0x7f024c0a96e0 2024-03-25T18:12:52.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.225+0000 7f0248ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:52.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.225+0000 7f0248ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:52.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.225+0000 7f0248ff9640 10 monclient: dump: 2024-03-25T18:12:52.227 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:52.227 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:52.227 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:52.227 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:52.227 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:52.227 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:52.227 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:52.227 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:52.227 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:52.227 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:52.227 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.225+0000 7f024a7fc640 4 set_mon_vals no callback set 2024-03-25T18:12:52.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.226+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f023c01a9d0 con 0x7f024c0a96e0 2024-03-25T18:12:52.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.227+0000 7f024b7fe640 10 monclient: get_auth_request con 0x7f023806e080 auth_method 0 2024-03-25T18:12:52.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.227+0000 7f0248ff9640 10 client.4210.objecter ms_handle_connect 0x7f023806e080 2024-03-25T18:12:52.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.228+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f023c054780 con 0x7f024c0a96e0 2024-03-25T18:12:52.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.228+0000 7f0248ff9640 10 client.4210.objecter ms_dispatch 0x7f024c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:52.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.228+0000 7f0248ff9640 3 client.4210.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:52.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.228+0000 7f0248ff9640 3 client.4210.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:52.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.228+0000 7f0248ff9640 20 client.4210.objecter dump_active .. 0 homeless 2024-03-25T18:12:52.241 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:09.981 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.290 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:10.046 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.348 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:10.105 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.403 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:10.163 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.445 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:10.205 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.455 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:10.213 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.459+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f023c0500c0 con 0x7f024c0a96e0 2024-03-25T18:12:52.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.459+0000 7f0248ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:52.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.459+0000 7f0248ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:52.489 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:10.231 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.496 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:10.247 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.516 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:10.257 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.579 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:10.335 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.638 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:10.395 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.645+0000 7f0252874640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:52.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.645+0000 7f0252874640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:52.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.645+0000 7f0252874640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:52.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.645+0000 7f0252874640 1 -- 172.21.15.63:0/2332750669 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f024c0b01d0 con 0x7f024c0a96e0 2024-03-25T18:12:52.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.660+0000 7f0248ff9640 1 -- 172.21.15.63:0/2332750669 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f023c0521c0 con 0x7f024c0a96e0 2024-03-25T18:12:52.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.660+0000 7f0248ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:52.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.660+0000 7f0248ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:52.662 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:12:52.662 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:52.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.663+0000 7f0252874640 10 librados: watch_flush enter 2024-03-25T18:12:52.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.663+0000 7f0252874640 10 librados: watch_flush exit 2024-03-25T18:12:52.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.663+0000 7f0252874640 20 client.4210.objecter shutdown clearing up homeless session... 2024-03-25T18:12:52.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.663+0000 7f0252874640 10 client.4210.objecter successfully canceled tick 2024-03-25T18:12:52.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.663+0000 7f0252874640 1 -- 172.21.15.63:0/2332750669 >> v1:172.21.15.81:6800/3640340873 conn(0x7f023806e080 legacy=0x7f0238070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:52.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.663+0000 7f0252874640 10 monclient: shutdown 2024-03-25T18:12:52.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.663+0000 7f0252874640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:52.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.663+0000 7f0252874640 1 -- 172.21.15.63:0/2332750669 >> v1:172.21.15.81:6789/0 conn(0x7f024c0a96e0 legacy=0x7f024c0af0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:52.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.663+0000 7f0252874640 1 -- 172.21.15.63:0/2332750669 shutdown_connections 2024-03-25T18:12:52.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.663+0000 7f0252874640 1 -- 172.21.15.63:0/2332750669 >> 172.21.15.63:0/2332750669 conn(0x7f024c0b7ea0 msgr2=0x7f024c0b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:52.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.663+0000 7f0252874640 1 -- 172.21.15.63:0/2332750669 shutdown_connections 2024-03-25T18:12:52.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.663+0000 7f0252874640 1 -- 172.21.15.63:0/2332750669 wait complete. 2024-03-25T18:12:52.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:52.663+0000 7f0252874640 1 librados: shutdown 2024-03-25T18:12:52.673 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:52.725 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:10.483 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.782 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:10.525 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.793 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:10.533 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.804 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:10.555 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.892 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:10.652 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.936 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:10.693 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.938 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:10.699 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.954 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:10.714 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.993 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:10.753 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:52.993 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:10.751 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:53.083 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:10.826 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:53.102 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:10.857 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:53.107 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:10.858 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:53.156 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:10.912 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:53.260 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:11.018 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:53.267 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:11.008 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:53.323 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:11.064 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:53.372 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:11.115 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:53.466 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:11.222 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:53.520 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:11.277 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:53.684 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:11.434 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:53.759 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:11.517 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:53.784 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:11.534 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:53.815 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:11.573 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:53.933 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:11.676 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:54.028 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:11.770 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:58.745 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:16.505 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:58.753 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:16.493 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:59.093 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:16.853 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:59.093 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:16.849 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:59.201 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:16.951 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:59.294 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:17.052 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:59.405 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:17.148 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:59.480 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:59.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.569+0000 7f3ebf98d640 10 monclient: get_monmap_and_config 2024-03-25T18:12:59.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.569+0000 7f3ebf98d640 10 monclient: build_initial_monmap 2024-03-25T18:12:59.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.569+0000 7f3ebf98d640 10 monclient: monmap: 2024-03-25T18:12:59.570 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:12:59.570 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:59.570 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:12:59.570259+0000 2024-03-25T18:12:59.570 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:12:59.570259+0000 2024-03-25T18:12:59.570 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:59.570 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:59.570 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:59.570 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:12:59.570 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:12:59.570 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.569+0000 7f3ebf98d640 1 Processor -- start 2024-03-25T18:12:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.569+0000 7f3ebf98d640 1 -- start start 2024-03-25T18:12:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.569+0000 7f3ebf98d640 10 monclient: init 2024-03-25T18:12:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.570+0000 7f3ebf98d640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.570+0000 7f3ebf98d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.570+0000 7f3ebf98d640 10 monclient(hunting): picked mon.noname-b con 0x7f3eb80a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.570+0000 7f3ebf98d640 10 monclient(hunting): picked mon.noname-c con 0x7f3eb813f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.570+0000 7f3ebf98d640 10 monclient(hunting): picked mon.noname-a con 0x7f3eb81430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.570+0000 7f3ebf98d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb8142520 con 0x7f3eb81430c0 2024-03-25T18:12:59.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.570+0000 7f3ebf98d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb80b28a0 con 0x7f3eb813f710 2024-03-25T18:12:59.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.570+0000 7f3ebf98d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb80b2a30 con 0x7f3eb80a98f0 2024-03-25T18:12:59.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.570+0000 7f3ebf98d640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:59.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.570+0000 7f3ebf98d640 10 monclient(hunting): authenticate will time out at 7473.757096s 2024-03-25T18:12:59.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.571+0000 7f3ebd729640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f3eb80a98f0 0x7f3eb80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:57698/0 (socket says 172.21.15.63:57698) 2024-03-25T18:12:59.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.571+0000 7f3ebd729640 1 -- 172.21.15.63:0/3245270326 learned_addr learned my addr 172.21.15.63:0/3245270326 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:12:59.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.574+0000 7f3eaffff640 1 -- 172.21.15.63:0/3245270326 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155666677 0 0) 0x7f3eb80b2a30 con 0x7f3eb80a98f0 2024-03-25T18:12:59.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.574+0000 7f3eaffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:59.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.574+0000 7f3eaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:59.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.574+0000 7f3eaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:59.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.574+0000 7f3eaffff640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:12:59.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.574+0000 7f3eaffff640 1 -- 172.21.15.63:0/3245270326 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ea4003520 con 0x7f3eb80a98f0 2024-03-25T18:12:59.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.574+0000 7f3eaffff640 1 -- 172.21.15.63:0/3245270326 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675433703 0 0) 0x7f3eb8142520 con 0x7f3eb81430c0 2024-03-25T18:12:59.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.574+0000 7f3eaffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:59.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.574+0000 7f3eaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:59.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.574+0000 7f3eaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:59.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.574+0000 7f3eaffff640 10 monclient(hunting): my global_id is 4203 2024-03-25T18:12:59.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.574+0000 7f3eaffff640 1 -- 172.21.15.63:0/3245270326 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eb80b2a30 con 0x7f3eb81430c0 2024-03-25T18:12:59.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.575+0000 7f3eaffff640 1 -- 172.21.15.63:0/3245270326 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187116860 0 0) 0x7f3eb80b28a0 con 0x7f3eb813f710 2024-03-25T18:12:59.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.575+0000 7f3eaffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:59.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.575+0000 7f3eaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:59.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.575+0000 7f3eaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:59.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.575+0000 7f3eaffff640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:12:59.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.575+0000 7f3eaffff640 1 -- 172.21.15.63:0/3245270326 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eb8142520 con 0x7f3eb813f710 2024-03-25T18:12:59.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.577+0000 7f3eaffff640 1 -- 172.21.15.63:0/3245270326 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 454646711 0 0) 0x7f3ea4003520 con 0x7f3eb80a98f0 2024-03-25T18:12:59.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.577+0000 7f3eaffff640 1 -- 172.21.15.63:0/3245270326 >> v1:172.21.15.81:6789/0 conn(0x7f3eb813f710 legacy=0x7f3eb81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:59.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.577+0000 7f3eaffff640 1 -- 172.21.15.63:0/3245270326 >> v1:172.21.15.63:6789/0 conn(0x7f3eb81430c0 legacy=0x7f3eb814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:59.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.577+0000 7f3eaffff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:59.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.577+0000 7f3eaffff640 1 monclient: found mon.noname-b 2024-03-25T18:12:59.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.577+0000 7f3eaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:59.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.577+0000 7f3eaffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:12:59.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.577+0000 7f3eaffff640 1 -- 172.21.15.63:0/3245270326 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb80b44c0 con 0x7f3eb80a98f0 2024-03-25T18:12:59.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.577+0000 7f3eaffff640 10 monclient: _finish_auth 0 2024-03-25T18:12:59.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.577+0000 7f3eaffff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:59.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.577+0000 7f3eaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:59.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.577+0000 7f3eaffff640 1 -- 172.21.15.63:0/3245270326 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3eb4001690 con 0x7f3eb80a98f0 2024-03-25T18:12:59.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.577+0000 7f3eaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:59.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.577+0000 7f3eaffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:59.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.577+0000 7f3eaffff640 10 monclient: dump: 2024-03-25T18:12:59.580 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:59.580 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:59.580 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:59.581 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:59.581 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:59.581 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:59.581 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:59.581 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:59.581 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:59.581 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:59.581 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.578+0000 7f3ebf98d640 5 monclient: authenticate success, global_id 4214 2024-03-25T18:12:59.581 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.578+0000 7f3ebf98d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:59.582 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.579+0000 7f3eaffff640 1 -- 172.21.15.63:0/3245270326 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3eb4001870 con 0x7f3eb80a98f0 2024-03-25T18:12:59.582 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.579+0000 7f3eaffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:59.582 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.579+0000 7f3ebf98d640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:59.582 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.579+0000 7f3ebf98d640 4 set_mon_vals no callback set 2024-03-25T18:12:59.582 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.580+0000 7f3eaffff640 1 -- 172.21.15.63:0/3245270326 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3eb4001690 con 0x7f3eb80a98f0 2024-03-25T18:12:59.582 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.580+0000 7f3eaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:59.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.580+0000 7f3eaffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:59.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.580+0000 7f3eaffff640 10 monclient: dump: 2024-03-25T18:12:59.583 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:59.583 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:59.583 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:59.583 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:59.583 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:59.583 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:59.583 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:59.583 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:59.583 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:59.583 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:59.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.580+0000 7f3ebf98d640 10 monclient: shutdown 2024-03-25T18:12:59.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.580+0000 7f3ebf98d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:59.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.580+0000 7f3ebf98d640 1 -- 172.21.15.63:0/3245270326 >> v1:172.21.15.63:6790/0 conn(0x7f3eb80a98f0 legacy=0x7f3eb80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:59.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.580+0000 7f3ebf98d640 1 -- 172.21.15.63:0/3245270326 shutdown_connections 2024-03-25T18:12:59.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.580+0000 7f3ebf98d640 1 -- 172.21.15.63:0/3245270326 >> 172.21.15.63:0/3245270326 conn(0x7f3eb80b7ea0 msgr2=0x7f3eb80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:59.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.580+0000 7f3ebf98d640 1 -- 172.21.15.63:0/3245270326 shutdown_connections 2024-03-25T18:12:59.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.581+0000 7f3ebf98d640 1 -- 172.21.15.63:0/3245270326 wait complete. 2024-03-25T18:12:59.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.581+0000 7f3ebf98d640 10 monclient: build_initial_monmap 2024-03-25T18:12:59.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.581+0000 7f3ebf98d640 10 monclient: monmap: 2024-03-25T18:12:59.584 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:12:59.584 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:59.584 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:12:59.584 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:12:59.584 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:59.584 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:12:59.584 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:12:59.584 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:12:59.585 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:12:59.585 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:59.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.582+0000 7f3ebf98d640 1 librados: starting msgr at 2024-03-25T18:12:59.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.582+0000 7f3ebf98d640 1 librados: starting objecter 2024-03-25T18:12:59.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.582+0000 7f3ebf98d640 1 Processor -- start 2024-03-25T18:12:59.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.582+0000 7f3ebf98d640 1 -- start start 2024-03-25T18:12:59.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.582+0000 7f3ebf98d640 1 librados: setting wanted keys 2024-03-25T18:12:59.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.582+0000 7f3ebf98d640 1 librados: calling monclient init 2024-03-25T18:12:59.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.582+0000 7f3ebf98d640 10 monclient: init 2024-03-25T18:12:59.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.583+0000 7f3ebf98d640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:59.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.583+0000 7f3ebf98d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:59.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.583+0000 7f3ebf98d640 10 monclient(hunting): picked mon.noname-b con 0x7f3eb80a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:12:59.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.583+0000 7f3ebf98d640 10 monclient(hunting): picked mon.noname-a con 0x7f3eb813f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:12:59.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.583+0000 7f3ebf98d640 10 monclient(hunting): picked mon.noname-c con 0x7f3eb81430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:12:59.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.583+0000 7f3ebf98d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb8139680 con 0x7f3eb813f710 2024-03-25T18:12:59.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.583+0000 7f3ebf98d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb8138cf0 con 0x7f3eb80a98f0 2024-03-25T18:12:59.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.583+0000 7f3ebf98d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb8136620 con 0x7f3eb81430c0 2024-03-25T18:12:59.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.583+0000 7f3ebf98d640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:59.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.583+0000 7f3ebf98d640 10 monclient(hunting): authenticate will time out at 7473.770026s 2024-03-25T18:12:59.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.584+0000 7f3ebcf28640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f3eb813f710 0x7f3eb81dabb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:57756/0 (socket says 172.21.15.63:57756) 2024-03-25T18:12:59.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.584+0000 7f3ebcf28640 1 -- 172.21.15.63:0/3813930690 learned_addr learned my addr 172.21.15.63:0/3813930690 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:12:59.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.584+0000 7f3eadffb640 10 client.?.objecter ms_handle_connect 0x7f3eb813f710 2024-03-25T18:12:59.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.584+0000 7f3eadffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:59.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.584+0000 7f3eadffb640 10 client.?.objecter ms_handle_connect 0x7f3eb80a98f0 2024-03-25T18:12:59.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.584+0000 7f3eadffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:59.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.585+0000 7f3eadffb640 10 client.?.objecter ms_handle_connect 0x7f3eb81430c0 2024-03-25T18:12:59.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.585+0000 7f3eadffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:59.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.586+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483095978 0 0) 0x7f3eb8139680 con 0x7f3eb813f710 2024-03-25T18:12:59.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.586+0000 7f3eadffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:59.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.586+0000 7f3eadffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:59.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.586+0000 7f3eadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:59.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.586+0000 7f3eadffb640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:12:59.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.586+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e9c004eb0 con 0x7f3eb813f710 2024-03-25T18:12:59.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.587+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721351712 0 0) 0x7f3eb8138cf0 con 0x7f3eb80a98f0 2024-03-25T18:12:59.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.587+0000 7f3eadffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:59.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.587+0000 7f3eadffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:59.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.587+0000 7f3eadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:59.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.587+0000 7f3eadffb640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:12:59.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.587+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eb8139680 con 0x7f3eb80a98f0 2024-03-25T18:12:59.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.587+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753670145 0 0) 0x7f3eb8136620 con 0x7f3eb81430c0 2024-03-25T18:12:59.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.587+0000 7f3eadffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:59.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.587+0000 7f3eadffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:59.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.587+0000 7f3eadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:59.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.587+0000 7f3eadffb640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:12:59.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.587+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eb8138cf0 con 0x7f3eb81430c0 2024-03-25T18:12:59.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.593+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1854379486 0 0) 0x7f3eb8139680 con 0x7f3eb80a98f0 2024-03-25T18:12:59.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.593+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3eb8136620 con 0x7f3eb80a98f0 2024-03-25T18:12:59.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.593+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3eb4001bc0 con 0x7f3eb80a98f0 2024-03-25T18:12:59.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.593+0000 7f3eadffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:59.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.593+0000 7f3eadffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:59.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.593+0000 7f3eadffb640 10 monclient(hunting): dump: 2024-03-25T18:12:59.594 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:59.594 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:59.594 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:59.594 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:59.595 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:59.595 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:59.595 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:59.595 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:59.595 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:59.595 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:59.595 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.593+0000 7f3eadffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:59.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.594+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2976848090 0 0) 0x7f3eb8138cf0 con 0x7f3eb81430c0 2024-03-25T18:12:59.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.594+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3eb8139680 con 0x7f3eb81430c0 2024-03-25T18:12:59.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.594+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3eb0001a30 con 0x7f3eb81430c0 2024-03-25T18:12:59.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.594+0000 7f3eadffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:59.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.594+0000 7f3eadffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:59.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.594+0000 7f3eadffb640 10 monclient(hunting): dump: 2024-03-25T18:12:59.596 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:59.596 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:59.596 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:59.596 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:59.596 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:59.596 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:59.596 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:59.596 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:59.596 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:59.597 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:59.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.594+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2827594413 0 0) 0x7f3e9c004eb0 con 0x7f3eb813f710 2024-03-25T18:12:59.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.594+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3eb8138cf0 con 0x7f3eb813f710 2024-03-25T18:12:59.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.594+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3ea8001a10 con 0x7f3eb813f710 2024-03-25T18:12:59.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.594+0000 7f3eadffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:59.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.594+0000 7f3eadffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:59.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.594+0000 7f3eadffb640 10 monclient(hunting): dump: 2024-03-25T18:12:59.597 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:59.597 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:59.597 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:59.597 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:59.597 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:59.597 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:59.597 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:59.597 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:59.598 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:59.598 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:59.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 568494828 0 0) 0x7f3eb8136620 con 0x7f3eb80a98f0 2024-03-25T18:12:59.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 >> v1:172.21.15.63:6790/0 conn(0x7f3eb81430c0 legacy=0x7f3eb81e2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:59.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 >> v1:172.21.15.63:6789/0 conn(0x7f3eb813f710 legacy=0x7f3eb81dabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:59.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3eadffb640 10 monclient: _finish_hunting 0 2024-03-25T18:12:59.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3eadffb640 1 monclient: found mon.b 2024-03-25T18:12:59.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3eadffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:59.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3eadffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:59.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb80b1620 con 0x7f3eb80a98f0 2024-03-25T18:12:59.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3eadffb640 10 monclient: _finish_auth 0 2024-03-25T18:12:59.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3eadffb640 10 monclient: _check_auth_tickets 2024-03-25T18:12:59.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3eadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:59.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3ebf98d640 5 monclient: authenticate success, global_id 4216 2024-03-25T18:12:59.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3ebf98d640 10 monclient: _renew_subs 2024-03-25T18:12:59.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3ebf98d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:59.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3ebf98d640 1 -- 172.21.15.63:0/3813930690 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb8139110 con 0x7f3eb80a98f0 2024-03-25T18:12:59.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3ebf98d640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:59.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3ebf98d640 10 monclient: _renew_subs 2024-03-25T18:12:59.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3ebf98d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:59.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3ebf98d640 1 -- 172.21.15.63:0/3813930690 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb813b770 con 0x7f3eb80a98f0 2024-03-25T18:12:59.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.597+0000 7f3ebf98d640 1 librados: init done 2024-03-25T18:12:59.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.599+0000 7f3ebf98d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:59.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.599+0000 7f3ebf98d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:59.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.599+0000 7f3ebf98d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:12:59.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.599+0000 7f3ebf98d640 1 -- 172.21.15.63:0/3813930690 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb8139460 con 0x7f3eb80a98f0 2024-03-25T18:12:59.602 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:17.359 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:12:59.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.620+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3eb4004a90 con 0x7f3eb80a98f0 2024-03-25T18:12:59.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.621+0000 7f3eadffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:59.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.621+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3eb4004da0 con 0x7f3eb80a98f0 2024-03-25T18:12:59.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.621+0000 7f3eadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:59.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.621+0000 7f3eadffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:59.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.621+0000 7f3eadffb640 10 monclient: dump: 2024-03-25T18:12:59.624 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:12:59.624 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:12:59.624 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:59.624 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:12:59.624 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:59.624 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:12:59.624 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:12:59.624 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:12:59.624 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:12:59.624 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:12:59.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.621+0000 7f3eaf7fe640 4 set_mon_vals no callback set 2024-03-25T18:12:59.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.622+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f3eb401ad70 con 0x7f3eb80a98f0 2024-03-25T18:12:59.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.623+0000 7f3ebcf28640 10 monclient: get_auth_request con 0x7f3e9c06e290 auth_method 0 2024-03-25T18:12:59.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.623+0000 7f3eadffb640 10 client.4216.objecter ms_handle_connect 0x7f3e9c06e290 2024-03-25T18:12:59.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.626+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f3eb4054850 con 0x7f3eb80a98f0 2024-03-25T18:12:59.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.626+0000 7f3eadffb640 10 client.4216.objecter ms_dispatch 0x7f3eb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:59.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.626+0000 7f3eadffb640 3 client.4216.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:59.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.626+0000 7f3eadffb640 3 client.4216.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:59.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.626+0000 7f3eadffb640 20 client.4216.objecter dump_active .. 0 homeless 2024-03-25T18:12:59.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.983+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3eb4050360 con 0x7f3eb80a98f0 2024-03-25T18:12:59.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.983+0000 7f3eadffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:59.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:12:59.983+0000 7f3eadffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:00.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.220+0000 7f3ebf98d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:00.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.220+0000 7f3ebf98d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:00.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.220+0000 7f3ebf98d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:13:00.222 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.220+0000 7f3ebf98d640 1 -- 172.21.15.63:0/3813930690 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3eb80627b0 con 0x7f3eb80a98f0 2024-03-25T18:13:00.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.243+0000 7f3eadffb640 1 -- 172.21.15.63:0/3813930690 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f3eb4052290 con 0x7f3eb80a98f0 2024-03-25T18:13:00.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.243+0000 7f3eadffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:00.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.243+0000 7f3eadffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:00.244 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:13:00.244 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:00.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.245+0000 7f3ebf98d640 10 librados: watch_flush enter 2024-03-25T18:13:00.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.245+0000 7f3ebf98d640 10 librados: watch_flush exit 2024-03-25T18:13:00.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.245+0000 7f3ebf98d640 20 client.4216.objecter shutdown clearing up homeless session... 2024-03-25T18:13:00.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.245+0000 7f3ebf98d640 10 client.4216.objecter successfully canceled tick 2024-03-25T18:13:00.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.245+0000 7f3ebf98d640 1 -- 172.21.15.63:0/3813930690 >> v1:172.21.15.81:6800/3640340873 conn(0x7f3e9c06e290 legacy=0x7f3e9c070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:00.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.245+0000 7f3ebf98d640 10 monclient: shutdown 2024-03-25T18:13:00.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.245+0000 7f3ebf98d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:00.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.245+0000 7f3ebf98d640 1 -- 172.21.15.63:0/3813930690 >> v1:172.21.15.81:6789/0 conn(0x7f3eb80a98f0 legacy=0x7f3eb81371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:00.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.245+0000 7f3ebf98d640 1 -- 172.21.15.63:0/3813930690 shutdown_connections 2024-03-25T18:13:00.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.245+0000 7f3ebf98d640 1 -- 172.21.15.63:0/3813930690 >> 172.21.15.63:0/3813930690 conn(0x7f3eb80b7ea0 msgr2=0x7f3eb8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:00.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.245+0000 7f3ebf98d640 1 -- 172.21.15.63:0/3813930690 shutdown_connections 2024-03-25T18:13:00.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.245+0000 7f3ebf98d640 1 -- 172.21.15.63:0/3813930690 wait complete. 2024-03-25T18:13:00.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:00.245+0000 7f3ebf98d640 1 librados: shutdown 2024-03-25T18:13:00.257 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:04.237 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:21.997 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:04.243 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:22.004 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:04.375 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:22.135 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:04.379 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:22.139 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:04.418 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:22.158 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:04.422 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:22.162 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:04.569 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:22.324 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:04.575 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:22.330 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:04.978 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:22.728 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:04.982 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:22.732 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:05.201 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:22.958 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:05.207 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:22.964 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:05.243 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:23.001 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:05.247 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:23.005 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:05.424 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:23.166 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:05.428 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:23.170 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:07.064 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:07.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.168+0000 7f7da35d0640 10 monclient: get_monmap_and_config 2024-03-25T18:13:07.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.168+0000 7f7da35d0640 10 monclient: build_initial_monmap 2024-03-25T18:13:07.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.168+0000 7f7da35d0640 10 monclient: monmap: 2024-03-25T18:13:07.170 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:13:07.170 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:07.170 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:13:07.169386+0000 2024-03-25T18:13:07.170 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:13:07.169386+0000 2024-03-25T18:13:07.170 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:07.170 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:07.170 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:13:07.170 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:13:07.170 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:13:07.170 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:07.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.168+0000 7f7da35d0640 1 Processor -- start 2024-03-25T18:13:07.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.169+0000 7f7da35d0640 1 -- start start 2024-03-25T18:13:07.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.169+0000 7f7da35d0640 10 monclient: init 2024-03-25T18:13:07.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.169+0000 7f7da35d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:07.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.169+0000 7f7da35d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:07.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.169+0000 7f7da35d0640 10 monclient(hunting): picked mon.noname-c con 0x7f7d9c0b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T18:13:07.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.169+0000 7f7da35d0640 10 monclient(hunting): picked mon.noname-a con 0x7f7d9c0b1730 addr v1:172.21.15.63:6789/0 2024-03-25T18:13:07.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.170+0000 7f7da35d0640 10 monclient(hunting): picked mon.noname-b con 0x7f7d9c14be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:13:07.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.170+0000 7f7da35d0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9c0af940 con 0x7f7d9c0b1730 2024-03-25T18:13:07.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.170+0000 7f7da35d0640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9c0b0a70 con 0x7f7d9c0b4d60 2024-03-25T18:13:07.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.170+0000 7f7da35d0640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9c0b0d50 con 0x7f7d9c14be90 2024-03-25T18:13:07.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.170+0000 7f7da35d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:07.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.170+0000 7f7da35d0640 10 monclient(hunting): authenticate will time out at 7481.356757s 2024-03-25T18:13:07.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.170+0000 7f7da0b6b640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f7d9c0b1730 0x7f7d9c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:57772/0 (socket says 172.21.15.63:57772) 2024-03-25T18:13:07.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.170+0000 7f7da0b6b640 1 -- 172.21.15.63:0/2664416748 learned_addr learned my addr 172.21.15.63:0/2664416748 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:13:07.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.173+0000 7f7d93fff640 1 -- 172.21.15.63:0/2664416748 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941335338 0 0) 0x7f7d9c0af940 con 0x7f7d9c0b1730 2024-03-25T18:13:07.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.173+0000 7f7d93fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:07.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.173+0000 7f7d93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:07.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.173+0000 7f7d93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:07.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.173+0000 7f7d93fff640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:13:07.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.173+0000 7f7d93fff640 1 -- 172.21.15.63:0/2664416748 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d88003520 con 0x7f7d9c0b1730 2024-03-25T18:13:07.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.174+0000 7f7d93fff640 1 -- 172.21.15.63:0/2664416748 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332701155 0 0) 0x7f7d9c0b0d50 con 0x7f7d9c14be90 2024-03-25T18:13:07.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.174+0000 7f7d93fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:07.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.174+0000 7f7d93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:07.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.174+0000 7f7d93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:07.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.174+0000 7f7d93fff640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:13:07.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.174+0000 7f7d93fff640 1 -- 172.21.15.63:0/2664416748 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d9c0af940 con 0x7f7d9c14be90 2024-03-25T18:13:07.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.174+0000 7f7d93fff640 1 -- 172.21.15.63:0/2664416748 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613660517 0 0) 0x7f7d9c0b0a70 con 0x7f7d9c0b4d60 2024-03-25T18:13:07.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.174+0000 7f7d93fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:07.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.174+0000 7f7d93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:07.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.174+0000 7f7d93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:07.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.174+0000 7f7d93fff640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:13:07.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.174+0000 7f7d93fff640 1 -- 172.21.15.63:0/2664416748 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d9c0b0d50 con 0x7f7d9c0b4d60 2024-03-25T18:13:07.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7d93fff640 1 -- 172.21.15.63:0/2664416748 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 600966755 0 0) 0x7f7d88003520 con 0x7f7d9c0b1730 2024-03-25T18:13:07.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7d93fff640 1 -- 172.21.15.63:0/2664416748 >> v1:172.21.15.63:6790/0 conn(0x7f7d9c14be90 legacy=0x7f7d9c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:07.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7d93fff640 1 -- 172.21.15.63:0/2664416748 >> v1:172.21.15.81:6789/0 conn(0x7f7d9c0b4d60 legacy=0x7f7d9c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:07.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7d93fff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:07.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7d93fff640 1 monclient: found mon.noname-a 2024-03-25T18:13:07.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7d93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:07.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7d93fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:13:07.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7d93fff640 1 -- 172.21.15.63:0/2664416748 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c0b1030 con 0x7f7d9c0b1730 2024-03-25T18:13:07.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7d93fff640 10 monclient: _finish_auth 0 2024-03-25T18:13:07.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7d93fff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:07.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7d93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:07.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7d93fff640 1 -- 172.21.15.63:0/2664416748 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7d8c001690 con 0x7f7d9c0b1730 2024-03-25T18:13:07.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7d93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:07.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7d93fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:07.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7d93fff640 10 monclient: dump: 2024-03-25T18:13:07.179 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:07.179 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:07.179 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:07.179 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:07.179 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:07.179 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:07.180 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:07.180 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:07.180 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:07.180 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:07.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7da35d0640 5 monclient: authenticate success, global_id 4209 2024-03-25T18:13:07.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.176+0000 7f7da35d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:07.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.178+0000 7f7d93fff640 1 -- 172.21.15.63:0/2664416748 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d8c0017f0 con 0x7f7d9c0b1730 2024-03-25T18:13:07.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.178+0000 7f7d93fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:07.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.178+0000 7f7d93fff640 1 -- 172.21.15.63:0/2664416748 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7d8c001c00 con 0x7f7d9c0b1730 2024-03-25T18:13:07.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.178+0000 7f7d93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:07.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.178+0000 7f7d93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:07.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.178+0000 7f7d93fff640 10 monclient: dump: 2024-03-25T18:13:07.183 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:07.183 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:07.183 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:07.183 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:07.183 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:07.183 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:07.183 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:07.183 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:07.183 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:07.183 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:07.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.179+0000 7f7da35d0640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:07.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.179+0000 7f7da35d0640 4 set_mon_vals no callback set 2024-03-25T18:13:07.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.179+0000 7f7da35d0640 10 monclient: shutdown 2024-03-25T18:13:07.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.179+0000 7f7da35d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:07.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.179+0000 7f7da35d0640 1 -- 172.21.15.63:0/2664416748 >> v1:172.21.15.63:6789/0 conn(0x7f7d9c0b1730 legacy=0x7f7d9c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:07.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.179+0000 7f7da35d0640 1 -- 172.21.15.63:0/2664416748 shutdown_connections 2024-03-25T18:13:07.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.180+0000 7f7da35d0640 1 -- 172.21.15.63:0/2664416748 >> 172.21.15.63:0/2664416748 conn(0x7f7d9c0617a0 msgr2=0x7f7d9c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:07.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.180+0000 7f7da35d0640 1 -- 172.21.15.63:0/2664416748 shutdown_connections 2024-03-25T18:13:07.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.180+0000 7f7da35d0640 1 -- 172.21.15.63:0/2664416748 wait complete. 2024-03-25T18:13:07.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.181+0000 7f7da35d0640 10 monclient: build_initial_monmap 2024-03-25T18:13:07.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.181+0000 7f7da35d0640 10 monclient: monmap: 2024-03-25T18:13:07.184 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:13:07.184 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:07.184 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:13:07.185 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:13:07.185 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:07.185 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:13:07.185 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:13:07.185 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:13:07.185 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:13:07.185 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:07.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.181+0000 7f7da35d0640 1 librados: starting msgr at 2024-03-25T18:13:07.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.181+0000 7f7da35d0640 1 librados: starting objecter 2024-03-25T18:13:07.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.181+0000 7f7da35d0640 1 Processor -- start 2024-03-25T18:13:07.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.182+0000 7f7da35d0640 1 -- start start 2024-03-25T18:13:07.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.182+0000 7f7da35d0640 1 librados: setting wanted keys 2024-03-25T18:13:07.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.182+0000 7f7da35d0640 1 librados: calling monclient init 2024-03-25T18:13:07.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.182+0000 7f7da35d0640 10 monclient: init 2024-03-25T18:13:07.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.182+0000 7f7da35d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:07.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.182+0000 7f7da35d0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:07.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.182+0000 7f7da35d0640 10 monclient(hunting): picked mon.noname-a con 0x7f7d9c0b1730 addr v1:172.21.15.63:6789/0 2024-03-25T18:13:07.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.182+0000 7f7da35d0640 10 monclient(hunting): picked mon.noname-b con 0x7f7d9c0b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T18:13:07.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.182+0000 7f7da35d0640 10 monclient(hunting): picked mon.noname-c con 0x7f7d9c14be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:13:07.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.182+0000 7f7da35d0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9c141ca0 con 0x7f7d9c0b1730 2024-03-25T18:13:07.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.183+0000 7f7da35d0640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9c141ef0 con 0x7f7d9c0b4d60 2024-03-25T18:13:07.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.183+0000 7f7da35d0640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9c1d5360 con 0x7f7d9c14be90 2024-03-25T18:13:07.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.183+0000 7f7da35d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:07.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.183+0000 7f7da35d0640 10 monclient(hunting): authenticate will time out at 7481.369584s 2024-03-25T18:13:07.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.183+0000 7f7da136c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f7d9c0b1730 0x7f7d9c1405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:57784/0 (socket says 172.21.15.63:57784) 2024-03-25T18:13:07.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.183+0000 7f7da136c640 1 -- 172.21.15.63:0/621782579 learned_addr learned my addr 172.21.15.63:0/621782579 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:13:07.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.183+0000 7f7d91ffb640 10 client.?.objecter ms_handle_connect 0x7f7d9c0b1730 2024-03-25T18:13:07.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.183+0000 7f7d91ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:07.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.184+0000 7f7d91ffb640 10 client.?.objecter ms_handle_connect 0x7f7d9c0b4d60 2024-03-25T18:13:07.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.184+0000 7f7d91ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:07.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.184+0000 7f7d91ffb640 10 client.?.objecter ms_handle_connect 0x7f7d9c14be90 2024-03-25T18:13:07.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.184+0000 7f7d91ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:07.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.186+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408162405 0 0) 0x7f7d9c141ca0 con 0x7f7d9c0b1730 2024-03-25T18:13:07.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.186+0000 7f7d91ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:07.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.186+0000 7f7d91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:07.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.186+0000 7f7d91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:07.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.186+0000 7f7d91ffb640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:13:07.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.186+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d80004eb0 con 0x7f7d9c0b1730 2024-03-25T18:13:07.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.186+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396774288 0 0) 0x7f7d9c141ef0 con 0x7f7d9c0b4d60 2024-03-25T18:13:07.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.186+0000 7f7d91ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:07.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.186+0000 7f7d91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:07.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.186+0000 7f7d91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:07.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.186+0000 7f7d91ffb640 10 monclient(hunting): my global_id is 4222 2024-03-25T18:13:07.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.186+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d9c141ca0 con 0x7f7d9c0b4d60 2024-03-25T18:13:07.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.187+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843729415 0 0) 0x7f7d9c1d5360 con 0x7f7d9c14be90 2024-03-25T18:13:07.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.187+0000 7f7d91ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:07.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.187+0000 7f7d91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:07.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.187+0000 7f7d91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:07.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.187+0000 7f7d91ffb640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:13:07.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.187+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d9c141ef0 con 0x7f7d9c14be90 2024-03-25T18:13:07.191 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.190+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 713049859 0 0) 0x7f7d80004eb0 con 0x7f7d9c0b1730 2024-03-25T18:13:07.191 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.190+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d9c1d5360 con 0x7f7d9c0b1730 2024-03-25T18:13:07.191 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.190+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7d98001a70 con 0x7f7d9c0b1730 2024-03-25T18:13:07.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.190+0000 7f7d91ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:07.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.190+0000 7f7d91ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:07.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.190+0000 7f7d91ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:07.192 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:07.192 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:07.192 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:07.192 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:07.192 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:07.192 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:07.192 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:07.192 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:07.192 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:07.192 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:07.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.190+0000 7f7d91ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:07.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.193+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3305143225 0 0) 0x7f7d9c141ca0 con 0x7f7d9c0b4d60 2024-03-25T18:13:07.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.193+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d80004eb0 con 0x7f7d9c0b4d60 2024-03-25T18:13:07.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.193+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7d8c001b00 con 0x7f7d9c0b4d60 2024-03-25T18:13:07.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.193+0000 7f7d91ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:07.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.193+0000 7f7d91ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:07.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.193+0000 7f7d91ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:07.195 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:07.195 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:07.195 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:07.195 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:07.195 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:07.195 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:07.195 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:07.195 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:07.195 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:07.195 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:07.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.193+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2407003609 0 0) 0x7f7d9c141ef0 con 0x7f7d9c14be90 2024-03-25T18:13:07.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.193+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d9c141ca0 con 0x7f7d9c14be90 2024-03-25T18:13:07.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.193+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7d94001b50 con 0x7f7d9c14be90 2024-03-25T18:13:07.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.193+0000 7f7d91ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:07.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.193+0000 7f7d91ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:07.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.193+0000 7f7d91ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:07.196 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:07.196 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:07.196 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:07.196 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:07.196 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:07.196 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:07.196 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:07.196 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:07.196 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:07.196 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:07.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1621889930 0 0) 0x7f7d9c1d5360 con 0x7f7d9c0b1730 2024-03-25T18:13:07.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 >> v1:172.21.15.63:6790/0 conn(0x7f7d9c14be90 legacy=0x7f7d9c1d7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:07.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 >> v1:172.21.15.81:6789/0 conn(0x7f7d9c0b4d60 legacy=0x7f7d9c134f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:07.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7d91ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:07.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7d91ffb640 1 monclient: found mon.a 2024-03-25T18:13:07.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7d91ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:07.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7d91ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:07.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c140d00 con 0x7f7d9c0b1730 2024-03-25T18:13:07.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7d91ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:07.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7d91ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:07.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7d91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:07.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7da35d0640 5 monclient: authenticate success, global_id 4212 2024-03-25T18:13:07.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7da35d0640 10 monclient: _renew_subs 2024-03-25T18:13:07.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7da35d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:07.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7da35d0640 1 -- 172.21.15.63:0/621782579 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d9c135650 con 0x7f7d9c0b1730 2024-03-25T18:13:07.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7da35d0640 10 client.4212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:07.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7da35d0640 10 monclient: _renew_subs 2024-03-25T18:13:07.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7da35d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:07.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7da35d0640 1 -- 172.21.15.63:0/621782579 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d9c1d8280 con 0x7f7d9c0b1730 2024-03-25T18:13:07.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.195+0000 7f7da35d0640 1 librados: init done 2024-03-25T18:13:07.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.197+0000 7f7d877fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:07.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.197+0000 7f7d877fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:07.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.197+0000 7f7d877fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:07.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.197+0000 7f7d877fe640 1 -- 172.21.15.63:0/621782579 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d7c005080 con 0x7f7d9c0b1730 2024-03-25T18:13:07.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.412+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d98001bd0 con 0x7f7d9c0b1730 2024-03-25T18:13:07.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.412+0000 7f7d91ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:07.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.412+0000 7f7d937fe640 4 set_mon_vals no callback set 2024-03-25T18:13:07.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.453+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7d98003a40 con 0x7f7d9c0b1730 2024-03-25T18:13:07.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.453+0000 7f7d91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:07.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.453+0000 7f7d91ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:07.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.453+0000 7f7d91ffb640 10 monclient: dump: 2024-03-25T18:13:07.454 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:07.454 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:07.454 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:07.454 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:07.454 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:07.454 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:07.454 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:07.454 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:07.454 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:07.455 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:07.461 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.460+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f7d98003c20 con 0x7f7d9c0b1730 2024-03-25T18:13:07.461 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.461+0000 7f7da0b6b640 10 monclient: get_auth_request con 0x7f7d8006e120 auth_method 0 2024-03-25T18:13:07.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.461+0000 7f7d91ffb640 10 client.4212.objecter ms_handle_connect 0x7f7d8006e120 2024-03-25T18:13:07.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.466+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f7d980548f0 con 0x7f7d9c0b1730 2024-03-25T18:13:07.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.466+0000 7f7d91ffb640 10 client.4212.objecter ms_dispatch 0x7f7d9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:07.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.466+0000 7f7d91ffb640 3 client.4212.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:07.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.466+0000 7f7d91ffb640 3 client.4212.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:07.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.467+0000 7f7d91ffb640 20 client.4212.objecter dump_active .. 0 homeless 2024-03-25T18:13:07.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.467+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7d98050400 con 0x7f7d9c0b1730 2024-03-25T18:13:07.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.467+0000 7f7d91ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:07.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.467+0000 7f7d91ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:07.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.744+0000 7f7d877fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:07.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.744+0000 7f7d877fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:07.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.744+0000 7f7d877fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:07.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.744+0000 7f7d877fe640 1 -- 172.21.15.63:0/621782579 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d7c005330 con 0x7f7d9c0b1730 2024-03-25T18:13:07.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.787+0000 7f7d91ffb640 1 -- 172.21.15.63:0/621782579 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f7d98052330 con 0x7f7d9c0b1730 2024-03-25T18:13:07.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.787+0000 7f7d91ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:07.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.787+0000 7f7d91ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:07.789 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:13:07.789 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:07.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.790+0000 7f7d877fe640 10 librados: watch_flush enter 2024-03-25T18:13:07.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.790+0000 7f7d877fe640 10 librados: watch_flush exit 2024-03-25T18:13:07.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.790+0000 7f7d877fe640 20 client.4212.objecter shutdown clearing up homeless session... 2024-03-25T18:13:07.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.790+0000 7f7d877fe640 10 client.4212.objecter successfully canceled tick 2024-03-25T18:13:07.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.790+0000 7f7d877fe640 1 -- 172.21.15.63:0/621782579 >> v1:172.21.15.81:6800/3640340873 conn(0x7f7d8006e120 legacy=0x7f7d800705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:07.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.790+0000 7f7d877fe640 10 monclient: shutdown 2024-03-25T18:13:07.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.790+0000 7f7d877fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:07.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.790+0000 7f7d877fe640 1 -- 172.21.15.63:0/621782579 >> v1:172.21.15.63:6789/0 conn(0x7f7d9c0b1730 legacy=0x7f7d9c1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:07.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.790+0000 7f7d877fe640 1 -- 172.21.15.63:0/621782579 shutdown_connections 2024-03-25T18:13:07.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.790+0000 7f7d877fe640 1 -- 172.21.15.63:0/621782579 >> 172.21.15.63:0/621782579 conn(0x7f7d9c0617a0 msgr2=0x7f7d9c13c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:07.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.790+0000 7f7d877fe640 1 -- 172.21.15.63:0/621782579 shutdown_connections 2024-03-25T18:13:07.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.790+0000 7f7d877fe640 1 -- 172.21.15.63:0/621782579 wait complete. 2024-03-25T18:13:07.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:07.790+0000 7f7d877fe640 1 librados: shutdown 2024-03-25T18:13:07.804 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:08.343 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:26.103 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:08.403 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:26.163 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:08.664 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:26.404 35501== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:08.917 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:26.672 35481== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:09.629 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:27.386 35480== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:09.846 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:27.596 35499== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:09.870 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:27.628 35498== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:10.182 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:27.925 35500== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:14.610 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:14.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.750+0000 7f2bfd354640 10 monclient: get_monmap_and_config 2024-03-25T18:13:14.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.750+0000 7f2bfd354640 10 monclient: build_initial_monmap 2024-03-25T18:13:14.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.750+0000 7f2bfd354640 10 monclient: monmap: 2024-03-25T18:13:14.751 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:13:14.752 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:14.752 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:13:14.751926+0000 2024-03-25T18:13:14.752 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:13:14.751926+0000 2024-03-25T18:13:14.752 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:14.752 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:14.752 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:13:14.752 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:13:14.752 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:13:14.752 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:14.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.751+0000 7f2bfd354640 1 Processor -- start 2024-03-25T18:13:14.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.751+0000 7f2bfd354640 1 -- start start 2024-03-25T18:13:14.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.751+0000 7f2bfd354640 10 monclient: init 2024-03-25T18:13:14.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.752+0000 7f2bfd354640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:14.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.752+0000 7f2bfd354640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:14.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.752+0000 7f2bfd354640 10 monclient(hunting): picked mon.noname-b con 0x7f2bf80b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T18:13:14.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.752+0000 7f2bfd354640 10 monclient(hunting): picked mon.noname-c con 0x7f2bf80b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:13:14.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.752+0000 7f2bfd354640 10 monclient(hunting): picked mon.noname-a con 0x7f2bf814be90 addr v1:172.21.15.63:6789/0 2024-03-25T18:13:14.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.752+0000 7f2bfd354640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf80af940 con 0x7f2bf814be90 2024-03-25T18:13:14.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.752+0000 7f2bfd354640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf80b0a70 con 0x7f2bf80b1730 2024-03-25T18:13:14.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.752+0000 7f2bfd354640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf80b0d50 con 0x7f2bf80b4d60 2024-03-25T18:13:14.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.752+0000 7f2bfd354640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:14.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.753+0000 7f2bfd354640 10 monclient(hunting): authenticate will time out at 7488.939293s 2024-03-25T18:13:14.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.753+0000 7f2bf6ffd640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f2bf80b4d60 0x7f2bf813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:53570/0 (socket says 172.21.15.63:53570) 2024-03-25T18:13:14.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.753+0000 7f2bf77fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f2bf814be90 0x7f2bf814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:37584/0 (socket says 172.21.15.63:37584) 2024-03-25T18:13:14.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.753+0000 7f2bf6ffd640 1 -- 172.21.15.63:0/4071217352 learned_addr learned my addr 172.21.15.63:0/4071217352 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:13:14.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.757+0000 7f2bf5ffb640 1 -- 172.21.15.63:0/4071217352 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203539052 0 0) 0x7f2bf80b0a70 con 0x7f2bf80b1730 2024-03-25T18:13:14.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.757+0000 7f2bf5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:14.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.757+0000 7f2bf5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:14.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.757+0000 7f2bf5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:14.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.757+0000 7f2bf5ffb640 10 monclient(hunting): my global_id is 4225 2024-03-25T18:13:14.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.757+0000 7f2bf5ffb640 1 -- 172.21.15.63:0/4071217352 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be0003520 con 0x7f2bf80b1730 2024-03-25T18:13:14.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.757+0000 7f2bf5ffb640 1 -- 172.21.15.63:0/4071217352 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841501156 0 0) 0x7f2bf80af940 con 0x7f2bf814be90 2024-03-25T18:13:14.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.757+0000 7f2bf5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:14.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.757+0000 7f2bf5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:14.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.757+0000 7f2bf5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:14.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.757+0000 7f2bf5ffb640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:13:14.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.757+0000 7f2bf5ffb640 1 -- 172.21.15.63:0/4071217352 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bf80b0a70 con 0x7f2bf814be90 2024-03-25T18:13:14.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.758+0000 7f2bf5ffb640 1 -- 172.21.15.63:0/4071217352 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647646576 0 0) 0x7f2bf80b0d50 con 0x7f2bf80b4d60 2024-03-25T18:13:14.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.758+0000 7f2bf5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:14.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.758+0000 7f2bf5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:14.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.758+0000 7f2bf5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:14.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.758+0000 7f2bf5ffb640 10 monclient(hunting): my global_id is 4226 2024-03-25T18:13:14.761 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.758+0000 7f2bf5ffb640 1 -- 172.21.15.63:0/4071217352 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bf80af940 con 0x7f2bf80b4d60 2024-03-25T18:13:14.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.762+0000 7f2bf5ffb640 1 -- 172.21.15.63:0/4071217352 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1018662597 0 0) 0x7f2bf80b0a70 con 0x7f2bf814be90 2024-03-25T18:13:14.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.762+0000 7f2bf5ffb640 1 -- 172.21.15.63:0/4071217352 >> v1:172.21.15.63:6790/0 conn(0x7f2bf80b4d60 legacy=0x7f2bf813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:14.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.762+0000 7f2bf5ffb640 1 -- 172.21.15.63:0/4071217352 >> v1:172.21.15.81:6789/0 conn(0x7f2bf80b1730 legacy=0x7f2bf80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:14.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.762+0000 7f2bf5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:14.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.762+0000 7f2bf5ffb640 1 monclient: found mon.noname-a 2024-03-25T18:13:14.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.762+0000 7f2bf5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:14.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.762+0000 7f2bf5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:13:14.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.762+0000 7f2bf5ffb640 1 -- 172.21.15.63:0/4071217352 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf80b1030 con 0x7f2bf814be90 2024-03-25T18:13:14.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.762+0000 7f2bf5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:14.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.762+0000 7f2bf5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:14.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.762+0000 7f2bf5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:14.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.762+0000 7f2bfd354640 5 monclient: authenticate success, global_id 4215 2024-03-25T18:13:14.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.762+0000 7f2bfd354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:14.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.765+0000 7f2bf5ffb640 1 -- 172.21.15.63:0/4071217352 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2bf00018e0 con 0x7f2bf814be90 2024-03-25T18:13:14.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.765+0000 7f2bf5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:14.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.765+0000 7f2bf5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:14.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.765+0000 7f2bf5ffb640 10 monclient: dump: 2024-03-25T18:13:14.769 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:14.769 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:14.769 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:14.769 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:14.769 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:14.769 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:14.769 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:14.769 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:14.769 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:14.769 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:14.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.765+0000 7f2bf5ffb640 1 -- 172.21.15.63:0/4071217352 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bf0001b30 con 0x7f2bf814be90 2024-03-25T18:13:14.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.765+0000 7f2bfd354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:14.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.765+0000 7f2bf5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:14.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.765+0000 7f2bfd354640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:14.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.765+0000 7f2bf5ffb640 1 -- 172.21.15.63:0/4071217352 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2bf0001f00 con 0x7f2bf814be90 2024-03-25T18:13:14.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.765+0000 7f2bfd354640 4 set_mon_vals no callback set 2024-03-25T18:13:14.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.765+0000 7f2bf5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:14.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.765+0000 7f2bf5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:14.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.765+0000 7f2bf5ffb640 10 monclient: dump: 2024-03-25T18:13:14.770 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:14.770 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:14.770 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:14.770 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:14.770 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:14.770 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:14.770 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:14.770 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:14.770 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:14.770 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:14.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.766+0000 7f2bfd354640 10 monclient: shutdown 2024-03-25T18:13:14.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.766+0000 7f2bfd354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:14.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.766+0000 7f2bfd354640 1 -- 172.21.15.63:0/4071217352 >> v1:172.21.15.63:6789/0 conn(0x7f2bf814be90 legacy=0x7f2bf814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:14.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.766+0000 7f2bfd354640 1 -- 172.21.15.63:0/4071217352 shutdown_connections 2024-03-25T18:13:14.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.766+0000 7f2bfd354640 1 -- 172.21.15.63:0/4071217352 >> 172.21.15.63:0/4071217352 conn(0x7f2bf80617a0 msgr2=0x7f2bf8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:14.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.767+0000 7f2bfd354640 1 -- 172.21.15.63:0/4071217352 shutdown_connections 2024-03-25T18:13:14.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.767+0000 7f2bfd354640 1 -- 172.21.15.63:0/4071217352 wait complete. 2024-03-25T18:13:14.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.768+0000 7f2bfd354640 10 monclient: build_initial_monmap 2024-03-25T18:13:14.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.768+0000 7f2bfd354640 10 monclient: monmap: 2024-03-25T18:13:14.771 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:13:14.771 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:14.771 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:13:14.771 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:13:14.771 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:14.772 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:13:14.772 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:13:14.772 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:13:14.772 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:13:14.772 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:14.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.768+0000 7f2bfd354640 1 librados: starting msgr at 2024-03-25T18:13:14.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.768+0000 7f2bfd354640 1 librados: starting objecter 2024-03-25T18:13:14.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.768+0000 7f2bfd354640 1 Processor -- start 2024-03-25T18:13:14.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.768+0000 7f2bfd354640 1 -- start start 2024-03-25T18:13:14.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.768+0000 7f2bfd354640 1 librados: setting wanted keys 2024-03-25T18:13:14.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.768+0000 7f2bfd354640 1 librados: calling monclient init 2024-03-25T18:13:14.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.768+0000 7f2bfd354640 10 monclient: init 2024-03-25T18:13:14.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.769+0000 7f2bfd354640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:14.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.769+0000 7f2bfd354640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:14.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.769+0000 7f2bfd354640 10 monclient(hunting): picked mon.noname-a con 0x7f2bf80b1730 addr v1:172.21.15.63:6789/0 2024-03-25T18:13:14.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.769+0000 7f2bfd354640 10 monclient(hunting): picked mon.noname-b con 0x7f2bf80b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T18:13:14.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.769+0000 7f2bfd354640 10 monclient(hunting): picked mon.noname-c con 0x7f2bf814be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:13:14.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.769+0000 7f2bfd354640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf813a740 con 0x7f2bf80b1730 2024-03-25T18:13:14.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.769+0000 7f2bfd354640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf8139830 con 0x7f2bf80b4d60 2024-03-25T18:13:14.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.769+0000 7f2bfd354640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf81d7910 con 0x7f2bf814be90 2024-03-25T18:13:14.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.769+0000 7f2bfd354640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:14.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.769+0000 7f2bfd354640 10 monclient(hunting): authenticate will time out at 7488.956087s 2024-03-25T18:13:14.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.770+0000 7f2bf77fe640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f2bf814be90 0x7f2bf813b5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:53578/0 (socket says 172.21.15.63:53578) 2024-03-25T18:13:14.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.770+0000 7f2bf77fe640 1 -- 172.21.15.63:0/3543112163 learned_addr learned my addr 172.21.15.63:0/3543112163 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:13:14.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.770+0000 7f2bdf7fe640 10 client.?.objecter ms_handle_connect 0x7f2bf814be90 2024-03-25T18:13:14.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.770+0000 7f2bdf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:14.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.771+0000 7f2bdf7fe640 10 client.?.objecter ms_handle_connect 0x7f2bf80b1730 2024-03-25T18:13:14.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.771+0000 7f2bdf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:14.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.771+0000 7f2bdf7fe640 10 client.?.objecter ms_handle_connect 0x7f2bf80b4d60 2024-03-25T18:13:14.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.771+0000 7f2bdf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:14.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146641535 0 0) 0x7f2bf81d7910 con 0x7f2bf814be90 2024-03-25T18:13:14.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:14.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:14.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:14.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 10 monclient(hunting): my global_id is 4229 2024-03-25T18:13:14.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be4004eb0 con 0x7f2bf814be90 2024-03-25T18:13:14.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419135141 0 0) 0x7f2bf813a740 con 0x7f2bf80b1730 2024-03-25T18:13:14.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:14.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:14.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:14.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:13:14.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bf81d7910 con 0x7f2bf80b1730 2024-03-25T18:13:14.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756206716 0 0) 0x7f2bf8139830 con 0x7f2bf80b4d60 2024-03-25T18:13:14.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:14.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:14.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:14.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 10 monclient(hunting): my global_id is 4228 2024-03-25T18:13:14.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.774+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be4005360 con 0x7f2bf80b4d60 2024-03-25T18:13:14.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.780+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2552229104 0 0) 0x7f2be4005360 con 0x7f2bf80b4d60 2024-03-25T18:13:14.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.780+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bf8139830 con 0x7f2bf80b4d60 2024-03-25T18:13:14.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.780+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2bec0017b0 con 0x7f2bf80b4d60 2024-03-25T18:13:14.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.780+0000 7f2bdf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:14.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.780+0000 7f2bdf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:14.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.780+0000 7f2bdf7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:14.781 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:14.781 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:14.781 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:14.781 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:14.781 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:14.782 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:14.782 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:14.782 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:14.782 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:14.782 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:14.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.780+0000 7f2bdf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:14.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.782+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3121382049 0 0) 0x7f2be4004eb0 con 0x7f2bf814be90 2024-03-25T18:13:14.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.782+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2be4005360 con 0x7f2bf814be90 2024-03-25T18:13:14.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.782+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2bf00018e0 con 0x7f2bf814be90 2024-03-25T18:13:14.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.782+0000 7f2bdf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:14.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.782+0000 7f2bdf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:14.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.782+0000 7f2bdf7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:14.783 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:14.783 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:14.783 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:14.783 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:14.783 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:14.783 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:14.783 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:14.783 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:14.783 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:14.784 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:14.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.782+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3061303390 0 0) 0x7f2bf81d7910 con 0x7f2bf80b1730 2024-03-25T18:13:14.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.782+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2be4004eb0 con 0x7f2bf80b1730 2024-03-25T18:13:14.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.782+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2be8001870 con 0x7f2bf80b1730 2024-03-25T18:13:14.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.782+0000 7f2bdf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:14.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.782+0000 7f2bdf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:14.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.782+0000 7f2bdf7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:14.785 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:14.785 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:14.785 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:14.785 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:14.785 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:14.785 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:14.785 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:14.785 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:14.785 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:14.785 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:14.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.784+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1340105601 0 0) 0x7f2bf8139830 con 0x7f2bf80b4d60 2024-03-25T18:13:14.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.784+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 >> v1:172.21.15.63:6790/0 conn(0x7f2bf814be90 legacy=0x7f2bf813b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:14.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.784+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 >> v1:172.21.15.63:6789/0 conn(0x7f2bf80b1730 legacy=0x7f2bf8139120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:14.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bdf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:14.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bdf7fe640 1 monclient: found mon.b 2024-03-25T18:13:14.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:14.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bdf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:13:14.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf81d8540 con 0x7f2bf80b4d60 2024-03-25T18:13:14.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bdf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:14.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bdf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:14.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:14.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bfd354640 5 monclient: authenticate success, global_id 4228 2024-03-25T18:13:14.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bfd354640 10 monclient: _renew_subs 2024-03-25T18:13:14.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bfd354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:13:14.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bfd354640 1 -- 172.21.15.63:0/3543112163 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf813add0 con 0x7f2bf80b4d60 2024-03-25T18:13:14.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bfd354640 10 client.4228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:14.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bfd354640 10 monclient: _renew_subs 2024-03-25T18:13:14.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bfd354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:13:14.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bfd354640 1 -- 172.21.15.63:0/3543112163 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf8143200 con 0x7f2bf80b4d60 2024-03-25T18:13:14.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.785+0000 7f2bfd354640 1 librados: init done 2024-03-25T18:13:14.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.786+0000 7f2bfd354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:14.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.786+0000 7f2bfd354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:14.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.786+0000 7f2bfd354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:13:14.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:14.786+0000 7f2bfd354640 1 -- 172.21.15.63:0/3543112163 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf8143550 con 0x7f2bf80b4d60 2024-03-25T18:13:15.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.000+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bec001cc0 con 0x7f2bf80b4d60 2024-03-25T18:13:15.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.000+0000 7f2bdf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:15.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.000+0000 7f2bf57fa640 4 set_mon_vals no callback set 2024-03-25T18:13:15.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.212+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2bec0037b0 con 0x7f2bf80b4d60 2024-03-25T18:13:15.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.212+0000 7f2bdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:15.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.212+0000 7f2bdf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:15.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.212+0000 7f2bdf7fe640 10 monclient: dump: 2024-03-25T18:13:15.213 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:15.214 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:15.214 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:15.214 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:15.214 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:15.214 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:15.214 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:15.214 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:15.214 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:15.214 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:15.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.213+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f2bec0039d0 con 0x7f2bf80b4d60 2024-03-25T18:13:15.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.214+0000 7f2bf6ffd640 10 monclient: get_auth_request con 0x7f2be406e270 auth_method 0 2024-03-25T18:13:15.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.214+0000 7f2bdf7fe640 10 client.4228.objecter ms_handle_connect 0x7f2be406e270 2024-03-25T18:13:15.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.218+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f2bec053590 con 0x7f2bf80b4d60 2024-03-25T18:13:15.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.218+0000 7f2bdf7fe640 10 client.4228.objecter ms_dispatch 0x7f2bf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:15.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.218+0000 7f2bdf7fe640 3 client.4228.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:15.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.218+0000 7f2bdf7fe640 3 client.4228.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:15.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.218+0000 7f2bdf7fe640 20 client.4228.objecter dump_active .. 0 homeless 2024-03-25T18:13:15.222 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.222+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2bec04f0a0 con 0x7f2bf80b4d60 2024-03-25T18:13:15.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.222+0000 7f2bdf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:15.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.222+0000 7f2bdf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:15.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.446+0000 7f2bfd354640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:15.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.446+0000 7f2bfd354640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:15.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.446+0000 7f2bfd354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:13:15.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.446+0000 7f2bfd354640 1 -- 172.21.15.63:0/3543112163 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2bf81d86d0 con 0x7f2bf80b4d60 2024-03-25T18:13:15.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.469+0000 7f2bdf7fe640 1 -- 172.21.15.63:0/3543112163 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f2bec050fd0 con 0x7f2bf80b4d60 2024-03-25T18:13:15.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.469+0000 7f2bdf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:15.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.469+0000 7f2bdf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:15.471 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:13:15.471 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:15.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.471+0000 7f2bfd354640 10 librados: watch_flush enter 2024-03-25T18:13:15.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.471+0000 7f2bfd354640 10 librados: watch_flush exit 2024-03-25T18:13:15.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.471+0000 7f2bfd354640 20 client.4228.objecter shutdown clearing up homeless session... 2024-03-25T18:13:15.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.471+0000 7f2bfd354640 10 client.4228.objecter successfully canceled tick 2024-03-25T18:13:15.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.471+0000 7f2bfd354640 1 -- 172.21.15.63:0/3543112163 >> v1:172.21.15.81:6800/3640340873 conn(0x7f2be406e270 legacy=0x7f2be4070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:15.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.471+0000 7f2bfd354640 10 monclient: shutdown 2024-03-25T18:13:15.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.471+0000 7f2bfd354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:15.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.471+0000 7f2bfd354640 1 -- 172.21.15.63:0/3543112163 >> v1:172.21.15.81:6789/0 conn(0x7f2bf80b4d60 legacy=0x7f2bf81d7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:15.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.471+0000 7f2bfd354640 1 -- 172.21.15.63:0/3543112163 shutdown_connections 2024-03-25T18:13:15.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.471+0000 7f2bfd354640 1 -- 172.21.15.63:0/3543112163 >> 172.21.15.63:0/3543112163 conn(0x7f2bf80617a0 msgr2=0x7f2bf80b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:15.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.472+0000 7f2bfd354640 1 -- 172.21.15.63:0/3543112163 shutdown_connections 2024-03-25T18:13:15.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.472+0000 7f2bfd354640 1 -- 172.21.15.63:0/3543112163 wait complete. 2024-03-25T18:13:15.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:15.472+0000 7f2bfd354640 1 librados: shutdown 2024-03-25T18:13:15.484 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:16.820 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-03-25T18:13:16.815+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:13:17.369 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-03-25T18:13:17.365+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:13:17.566 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-03-25T18:13:17.562+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:13:17.609 INFO:tasks.ceph.osd.3.smithi063.stderr:2024-03-25T18:13:17.605+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:13:18.593 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-03-25T18:13:18.589+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:13:18.994 INFO:tasks.ceph.osd.0.smithi063.stderr:2024-03-25T18:13:18.991+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:13:19.277 INFO:tasks.ceph.osd.1.smithi063.stderr:2024-03-25T18:13:19.273+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:13:19.445 INFO:tasks.ceph.osd.2.smithi063.stderr:2024-03-25T18:13:19.441+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:13:22.290 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:22.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.405+0000 7f89b92a8640 10 monclient: get_monmap_and_config 2024-03-25T18:13:22.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.405+0000 7f89b92a8640 10 monclient: build_initial_monmap 2024-03-25T18:13:22.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.405+0000 7f89b92a8640 10 monclient: monmap: 2024-03-25T18:13:22.406 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:13:22.406 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:22.407 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:13:22.406765+0000 2024-03-25T18:13:22.407 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:13:22.406765+0000 2024-03-25T18:13:22.407 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:22.407 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:22.407 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:13:22.407 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:13:22.407 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:13:22.407 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:22.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.406+0000 7f89b92a8640 1 Processor -- start 2024-03-25T18:13:22.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.406+0000 7f89b92a8640 1 -- start start 2024-03-25T18:13:22.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.406+0000 7f89b92a8640 10 monclient: init 2024-03-25T18:13:22.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.406+0000 7f89b92a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:22.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.406+0000 7f89b92a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:22.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.406+0000 7f89b92a8640 10 monclient(hunting): picked mon.noname-c con 0x7f89b40a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:13:22.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.406+0000 7f89b92a8640 10 monclient(hunting): picked mon.noname-a con 0x7f89b413f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:13:22.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.406+0000 7f89b92a8640 10 monclient(hunting): picked mon.noname-b con 0x7f89b41430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:13:22.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.406+0000 7f89b92a8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b4142520 con 0x7f89b413f710 2024-03-25T18:13:22.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.406+0000 7f89b92a8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b40b28a0 con 0x7f89b40a98f0 2024-03-25T18:13:22.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.406+0000 7f89b92a8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b40b2a30 con 0x7f89b41430c0 2024-03-25T18:13:22.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.406+0000 7f89b92a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:22.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.406+0000 7f89b92a8640 10 monclient(hunting): authenticate will time out at 7496.593240s 2024-03-25T18:13:22.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.407+0000 7f89b37fe640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f89b41430c0 0x7f89b414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:34134/0 (socket says 172.21.15.63:34134) 2024-03-25T18:13:22.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.407+0000 7f89b37fe640 1 -- 172.21.15.63:0/2304687104 learned_addr learned my addr 172.21.15.63:0/2304687104 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:13:22.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.468+0000 7f89b1ffb640 1 -- 172.21.15.63:0/2304687104 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531961595 0 0) 0x7f89b4142520 con 0x7f89b413f710 2024-03-25T18:13:22.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.468+0000 7f89b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:22.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.468+0000 7f89b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:22.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.468+0000 7f89b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:22.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.468+0000 7f89b1ffb640 10 monclient(hunting): my global_id is 4245 2024-03-25T18:13:22.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.468+0000 7f89b1ffb640 1 -- 172.21.15.63:0/2304687104 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f899c003520 con 0x7f89b413f710 2024-03-25T18:13:22.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.471+0000 7f89b1ffb640 1 -- 172.21.15.63:0/2304687104 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313367908 0 0) 0x7f89b40b2a30 con 0x7f89b41430c0 2024-03-25T18:13:22.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.471+0000 7f89b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:22.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.471+0000 7f89b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:22.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.471+0000 7f89b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:22.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.471+0000 7f89b1ffb640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:13:22.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.471+0000 7f89b1ffb640 1 -- 172.21.15.63:0/2304687104 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b4142520 con 0x7f89b41430c0 2024-03-25T18:13:22.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.480+0000 7f89b1ffb640 1 -- 172.21.15.63:0/2304687104 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398517590 0 0) 0x7f89b40b28a0 con 0x7f89b40a98f0 2024-03-25T18:13:22.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.480+0000 7f89b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:22.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.480+0000 7f89b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:22.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.480+0000 7f89b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:22.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.480+0000 7f89b1ffb640 10 monclient(hunting): my global_id is 4255 2024-03-25T18:13:22.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.480+0000 7f89b1ffb640 1 -- 172.21.15.63:0/2304687104 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b40b2a30 con 0x7f89b40a98f0 2024-03-25T18:13:22.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b1ffb640 1 -- 172.21.15.63:0/2304687104 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2871624862 0 0) 0x7f899c003520 con 0x7f89b413f710 2024-03-25T18:13:22.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b1ffb640 1 -- 172.21.15.63:0/2304687104 >> v1:172.21.15.63:6790/0 conn(0x7f89b41430c0 legacy=0x7f89b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:22.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b1ffb640 1 -- 172.21.15.63:0/2304687104 >> v1:172.21.15.81:6789/0 conn(0x7f89b40a98f0 legacy=0x7f89b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:22.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:22.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b1ffb640 1 monclient: found mon.noname-a 2024-03-25T18:13:22.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:22.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:13:22.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b1ffb640 1 -- 172.21.15.63:0/2304687104 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b40b44c0 con 0x7f89b413f710 2024-03-25T18:13:22.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:22.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:22.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:22.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b1ffb640 1 -- 172.21.15.63:0/2304687104 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f89a8001690 con 0x7f89b413f710 2024-03-25T18:13:22.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b92a8640 5 monclient: authenticate success, global_id 4245 2024-03-25T18:13:22.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b92a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:22.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:22.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:22.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b1ffb640 10 monclient: dump: 2024-03-25T18:13:22.491 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:22.491 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:22.491 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:22.491 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:22.491 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:22.491 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:22.491 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:22.491 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:22.491 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:22.491 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:22.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.489+0000 7f89b92a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:22.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.509+0000 7f89b1ffb640 1 -- 172.21.15.63:0/2304687104 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89a80017f0 con 0x7f89b413f710 2024-03-25T18:13:22.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.509+0000 7f89b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:22.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.509+0000 7f89b1ffb640 1 -- 172.21.15.63:0/2304687104 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f89a8001c00 con 0x7f89b413f710 2024-03-25T18:13:22.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.509+0000 7f89b92a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:22.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.509+0000 7f89b92a8640 4 set_mon_vals no callback set 2024-03-25T18:13:22.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.509+0000 7f89b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:22.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.509+0000 7f89b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:22.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.509+0000 7f89b1ffb640 10 monclient: dump: 2024-03-25T18:13:22.511 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:22.511 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:22.511 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:22.511 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:22.511 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:22.511 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:22.511 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:22.511 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:22.511 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:22.511 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:22.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.510+0000 7f89b92a8640 10 monclient: shutdown 2024-03-25T18:13:22.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.510+0000 7f89b92a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:22.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.510+0000 7f89b92a8640 1 -- 172.21.15.63:0/2304687104 >> v1:172.21.15.63:6789/0 conn(0x7f89b413f710 legacy=0x7f89b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:22.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.510+0000 7f89b92a8640 1 -- 172.21.15.63:0/2304687104 shutdown_connections 2024-03-25T18:13:22.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.510+0000 7f89b92a8640 1 -- 172.21.15.63:0/2304687104 >> 172.21.15.63:0/2304687104 conn(0x7f89b40b7ea0 msgr2=0x7f89b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:22.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.510+0000 7f89b92a8640 1 -- 172.21.15.63:0/2304687104 shutdown_connections 2024-03-25T18:13:22.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.510+0000 7f89b92a8640 1 -- 172.21.15.63:0/2304687104 wait complete. 2024-03-25T18:13:22.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.511+0000 7f89b92a8640 10 monclient: build_initial_monmap 2024-03-25T18:13:22.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.511+0000 7f89b92a8640 10 monclient: monmap: 2024-03-25T18:13:22.514 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:13:22.514 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:22.514 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:13:22.514 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:13:22.514 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:22.514 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:13:22.514 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:13:22.514 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:13:22.515 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:13:22.515 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:22.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.511+0000 7f89b92a8640 1 librados: starting msgr at 2024-03-25T18:13:22.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.511+0000 7f89b92a8640 1 librados: starting objecter 2024-03-25T18:13:22.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.512+0000 7f89b92a8640 1 Processor -- start 2024-03-25T18:13:22.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.512+0000 7f89b92a8640 1 -- start start 2024-03-25T18:13:22.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.512+0000 7f89b92a8640 1 librados: setting wanted keys 2024-03-25T18:13:22.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.512+0000 7f89b92a8640 1 librados: calling monclient init 2024-03-25T18:13:22.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.512+0000 7f89b92a8640 10 monclient: init 2024-03-25T18:13:22.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.512+0000 7f89b92a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:22.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.512+0000 7f89b92a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:22.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.512+0000 7f89b92a8640 10 monclient(hunting): picked mon.noname-b con 0x7f89b40a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:13:22.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.512+0000 7f89b92a8640 10 monclient(hunting): picked mon.noname-a con 0x7f89b413f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:13:22.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.512+0000 7f89b92a8640 10 monclient(hunting): picked mon.noname-c con 0x7f89b41430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:13:22.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.512+0000 7f89b92a8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b4135550 con 0x7f89b413f710 2024-03-25T18:13:22.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.512+0000 7f89b92a8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b4136b50 con 0x7f89b40a98f0 2024-03-25T18:13:22.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.512+0000 7f89b92a8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b4136e30 con 0x7f89b41430c0 2024-03-25T18:13:22.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.512+0000 7f89b92a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:22.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.512+0000 7f89b92a8640 10 monclient(hunting): authenticate will time out at 7496.699276s 2024-03-25T18:13:22.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.525+0000 7f89b27fc640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f89b413f710 0x7f89b41375a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:60672/0 (socket says 172.21.15.63:60672) 2024-03-25T18:13:22.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.525+0000 7f89b27fc640 1 -- 172.21.15.63:0/425595922 learned_addr learned my addr 172.21.15.63:0/425595922 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:13:22.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.526+0000 7f899b7fe640 10 client.?.objecter ms_handle_connect 0x7f89b413f710 2024-03-25T18:13:22.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.526+0000 7f899b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:22.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.532+0000 7f899b7fe640 10 client.?.objecter ms_handle_connect 0x7f89b40a98f0 2024-03-25T18:13:22.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.532+0000 7f899b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:22.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.535+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197954613 0 0) 0x7f89b4135550 con 0x7f89b413f710 2024-03-25T18:13:22.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.535+0000 7f899b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:22.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.535+0000 7f899b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:22.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.535+0000 7f899b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:22.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.535+0000 7f899b7fe640 10 monclient(hunting): my global_id is 4248 2024-03-25T18:13:22.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.535+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a0004eb0 con 0x7f89b413f710 2024-03-25T18:13:22.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.539+0000 7f899b7fe640 10 client.?.objecter ms_handle_connect 0x7f89b41430c0 2024-03-25T18:13:22.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.539+0000 7f899b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:22.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.548+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838373664 0 0) 0x7f89b4136b50 con 0x7f89b40a98f0 2024-03-25T18:13:22.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.548+0000 7f899b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:22.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.548+0000 7f899b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:22.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.548+0000 7f899b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:22.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.548+0000 7f899b7fe640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:13:22.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.548+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b4135550 con 0x7f89b40a98f0 2024-03-25T18:13:22.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.550+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2250395061 0 0) 0x7f89a0004eb0 con 0x7f89b413f710 2024-03-25T18:13:22.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.550+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89b4136b50 con 0x7f89b413f710 2024-03-25T18:13:22.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.550+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f89a8001bc0 con 0x7f89b413f710 2024-03-25T18:13:22.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.550+0000 7f899b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:22.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.550+0000 7f899b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:22.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.550+0000 7f899b7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:22.551 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:22.551 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:22.552 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:22.552 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:22.552 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:22.552 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:22.552 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:22.552 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:22.552 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:22.552 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:22.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.550+0000 7f899b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:22.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.567+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282832031 0 0) 0x7f89b4136e30 con 0x7f89b41430c0 2024-03-25T18:13:22.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.567+0000 7f899b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:22.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.567+0000 7f899b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:22.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.567+0000 7f899b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:22.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.567+0000 7f899b7fe640 10 monclient(hunting): my global_id is 4259 2024-03-25T18:13:22.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.567+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a0004eb0 con 0x7f89b41430c0 2024-03-25T18:13:22.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4176233089 0 0) 0x7f89b4136b50 con 0x7f89b413f710 2024-03-25T18:13:22.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 >> v1:172.21.15.63:6790/0 conn(0x7f89b41430c0 legacy=0x7f89b41e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:22.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 >> v1:172.21.15.81:6789/0 conn(0x7f89b40a98f0 legacy=0x7f89b413b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:22.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f899b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:22.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f899b7fe640 1 monclient: found mon.a 2024-03-25T18:13:22.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f899b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:22.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f899b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:22.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b41357f0 con 0x7f89b413f710 2024-03-25T18:13:22.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f899b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:22.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f899b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:22.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f899b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:22.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f89b92a8640 5 monclient: authenticate success, global_id 4248 2024-03-25T18:13:22.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f89b92a8640 10 monclient: _renew_subs 2024-03-25T18:13:22.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f89b92a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:22.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f89b92a8640 1 -- 172.21.15.63:0/425595922 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b4137cb0 con 0x7f89b413f710 2024-03-25T18:13:22.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f89b92a8640 10 client.4248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:22.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f89b92a8640 10 monclient: _renew_subs 2024-03-25T18:13:22.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f89b92a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:22.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f89b92a8640 1 -- 172.21.15.63:0/425595922 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b41352e0 con 0x7f89b413f710 2024-03-25T18:13:22.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.573+0000 7f89b92a8640 1 librados: init done 2024-03-25T18:13:22.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.574+0000 7f89b92a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:22.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.574+0000 7f89b92a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:22.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.574+0000 7f89b92a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:22.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.574+0000 7f89b92a8640 1 -- 172.21.15.63:0/425595922 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b40627b0 con 0x7f89b413f710 2024-03-25T18:13:22.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.919+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89a80011f0 con 0x7f89b413f710 2024-03-25T18:13:22.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.919+0000 7f899b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:22.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.919+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f89a8003b00 con 0x7f89b413f710 2024-03-25T18:13:22.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.919+0000 7f899b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:22.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.919+0000 7f899b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:22.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.919+0000 7f899b7fe640 10 monclient: dump: 2024-03-25T18:13:22.921 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:22.921 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:22.921 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:22.921 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:22.921 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:22.922 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:22.922 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:22.922 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:22.922 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:22.922 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:22.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.919+0000 7f89b17fa640 4 set_mon_vals no callback set 2024-03-25T18:13:22.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.920+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f89a8003d20 con 0x7f89b413f710 2024-03-25T18:13:22.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.921+0000 7f89b2ffd640 10 monclient: get_auth_request con 0x7f89a006de00 auth_method 0 2024-03-25T18:13:22.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.921+0000 7f899b7fe640 10 client.4248.objecter ms_handle_connect 0x7f89a006de00 2024-03-25T18:13:22.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.928+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f89a8054880 con 0x7f89b413f710 2024-03-25T18:13:22.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.928+0000 7f899b7fe640 10 client.4248.objecter ms_dispatch 0x7f89b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:22.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.928+0000 7f899b7fe640 3 client.4248.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:22.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.928+0000 7f899b7fe640 3 client.4248.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:22.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.928+0000 7f899b7fe640 20 client.4248.objecter dump_active .. 0 homeless 2024-03-25T18:13:22.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.928+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f89a8050390 con 0x7f89b413f710 2024-03-25T18:13:22.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.928+0000 7f899b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:22.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:22.928+0000 7f899b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:23.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.186+0000 7f89b92a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:23.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.186+0000 7f89b92a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:23.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.186+0000 7f89b92a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:23.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.186+0000 7f89b92a8640 1 -- 172.21.15.63:0/425595922 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89b4135980 con 0x7f89b413f710 2024-03-25T18:13:23.231 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.231+0000 7f899b7fe640 1 -- 172.21.15.63:0/425595922 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f89a80522c0 con 0x7f89b413f710 2024-03-25T18:13:23.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.231+0000 7f899b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:23.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.231+0000 7f899b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:23.232 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:13:23.232 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:23.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.233+0000 7f89b92a8640 10 librados: watch_flush enter 2024-03-25T18:13:23.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.233+0000 7f89b92a8640 10 librados: watch_flush exit 2024-03-25T18:13:23.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.233+0000 7f89b92a8640 20 client.4248.objecter shutdown clearing up homeless session... 2024-03-25T18:13:23.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.233+0000 7f89b92a8640 10 client.4248.objecter successfully canceled tick 2024-03-25T18:13:23.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.233+0000 7f89b92a8640 1 -- 172.21.15.63:0/425595922 >> v1:172.21.15.81:6800/3640340873 conn(0x7f89a006de00 legacy=0x7f89a00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:23.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.233+0000 7f89b92a8640 10 monclient: shutdown 2024-03-25T18:13:23.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.233+0000 7f89b92a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:23.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.233+0000 7f89b92a8640 1 -- 172.21.15.63:0/425595922 >> v1:172.21.15.63:6789/0 conn(0x7f89b413f710 legacy=0x7f89b41375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:23.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.233+0000 7f89b92a8640 1 -- 172.21.15.63:0/425595922 shutdown_connections 2024-03-25T18:13:23.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.233+0000 7f89b92a8640 1 -- 172.21.15.63:0/425595922 >> 172.21.15.63:0/425595922 conn(0x7f89b40b7ea0 msgr2=0x7f89b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:23.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.233+0000 7f89b92a8640 1 -- 172.21.15.63:0/425595922 shutdown_connections 2024-03-25T18:13:23.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.233+0000 7f89b92a8640 1 -- 172.21.15.63:0/425595922 wait complete. 2024-03-25T18:13:23.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:23.233+0000 7f89b92a8640 1 librados: shutdown 2024-03-25T18:13:23.247 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:30.052 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:30.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.166+0000 7f802393d640 10 monclient: get_monmap_and_config 2024-03-25T18:13:30.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.166+0000 7f802393d640 10 monclient: build_initial_monmap 2024-03-25T18:13:30.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.166+0000 7f802393d640 10 monclient: monmap: 2024-03-25T18:13:30.167 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:13:30.168 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:30.168 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:13:30.167706+0000 2024-03-25T18:13:30.168 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:13:30.167706+0000 2024-03-25T18:13:30.168 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:30.168 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:30.168 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:13:30.168 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:13:30.168 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:13:30.168 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:30.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.167+0000 7f802393d640 1 Processor -- start 2024-03-25T18:13:30.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.167+0000 7f802393d640 1 -- start start 2024-03-25T18:13:30.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.167+0000 7f802393d640 10 monclient: init 2024-03-25T18:13:30.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.168+0000 7f802393d640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:30.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.168+0000 7f802393d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:30.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.168+0000 7f802393d640 10 monclient(hunting): picked mon.noname-c con 0x7f801c0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:13:30.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.168+0000 7f802393d640 10 monclient(hunting): picked mon.noname-b con 0x7f801c13f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:13:30.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.168+0000 7f802393d640 10 monclient(hunting): picked mon.noname-a con 0x7f801c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:13:30.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.168+0000 7f802393d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c142520 con 0x7f801c1430c0 2024-03-25T18:13:30.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.168+0000 7f802393d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c0b28a0 con 0x7f801c0a98f0 2024-03-25T18:13:30.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.168+0000 7f802393d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c0b2a30 con 0x7f801c13f710 2024-03-25T18:13:30.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.168+0000 7f802393d640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:30.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.168+0000 7f802393d640 10 monclient(hunting): authenticate will time out at 7504.355093s 2024-03-25T18:13:30.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.175+0000 7f8021eda640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f801c1430c0 0x7f801c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:54044/0 (socket says 172.21.15.63:54044) 2024-03-25T18:13:30.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.175+0000 7f8021eda640 1 -- 172.21.15.63:0/1140117403 learned_addr learned my addr 172.21.15.63:0/1140117403 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:13:30.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.191+0000 7f8013fff640 1 -- 172.21.15.63:0/1140117403 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198770835 0 0) 0x7f801c142520 con 0x7f801c1430c0 2024-03-25T18:13:30.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.191+0000 7f8013fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:30.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.191+0000 7f8013fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:30.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.191+0000 7f8013fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:30.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.191+0000 7f8013fff640 10 monclient(hunting): my global_id is 4251 2024-03-25T18:13:30.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.191+0000 7f8013fff640 1 -- 172.21.15.63:0/1140117403 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8008003520 con 0x7f801c1430c0 2024-03-25T18:13:30.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.204+0000 7f8013fff640 1 -- 172.21.15.63:0/1140117403 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222530421 0 0) 0x7f801c0b28a0 con 0x7f801c0a98f0 2024-03-25T18:13:30.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.204+0000 7f8013fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:30.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.204+0000 7f8013fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:30.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.204+0000 7f8013fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:30.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.204+0000 7f8013fff640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:13:30.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.204+0000 7f8013fff640 1 -- 172.21.15.63:0/1140117403 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f801c142520 con 0x7f801c0a98f0 2024-03-25T18:13:30.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.205+0000 7f8013fff640 1 -- 172.21.15.63:0/1140117403 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235610499 0 0) 0x7f801c0b2a30 con 0x7f801c13f710 2024-03-25T18:13:30.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.205+0000 7f8013fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:30.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.205+0000 7f8013fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:30.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.205+0000 7f8013fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:30.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.205+0000 7f8013fff640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:13:30.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.205+0000 7f8013fff640 1 -- 172.21.15.63:0/1140117403 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f801c0b28a0 con 0x7f801c13f710 2024-03-25T18:13:30.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f8013fff640 1 -- 172.21.15.63:0/1140117403 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1265535456 0 0) 0x7f801c142520 con 0x7f801c0a98f0 2024-03-25T18:13:30.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f8013fff640 1 -- 172.21.15.63:0/1140117403 >> v1:172.21.15.63:6790/0 conn(0x7f801c13f710 legacy=0x7f801c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:30.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f8013fff640 1 -- 172.21.15.63:0/1140117403 >> v1:172.21.15.63:6789/0 conn(0x7f801c1430c0 legacy=0x7f801c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:30.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f8013fff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:30.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f8013fff640 1 monclient: found mon.noname-c 2024-03-25T18:13:30.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f8013fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:30.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f8013fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:13:30.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f8013fff640 1 -- 172.21.15.63:0/1140117403 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801c0b44c0 con 0x7f801c0a98f0 2024-03-25T18:13:30.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f8013fff640 10 monclient: _finish_auth 0 2024-03-25T18:13:30.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f8013fff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:30.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f8013fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:30.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f8013fff640 1 -- 172.21.15.63:0/1140117403 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8018001670 con 0x7f801c0a98f0 2024-03-25T18:13:30.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f802393d640 5 monclient: authenticate success, global_id 4261 2024-03-25T18:13:30.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f802393d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:30.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f8013fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:30.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f8013fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:30.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f8013fff640 10 monclient: dump: 2024-03-25T18:13:30.217 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:30.217 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:30.217 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:30.218 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:30.218 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:30.218 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:30.218 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:30.218 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:30.218 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:30.218 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:30.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.215+0000 7f802393d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:30.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.222+0000 7f8013fff640 1 -- 172.21.15.63:0/1140117403 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80180017d0 con 0x7f801c0a98f0 2024-03-25T18:13:30.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.222+0000 7f8013fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:30.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.222+0000 7f8013fff640 1 -- 172.21.15.63:0/1140117403 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8018001be0 con 0x7f801c0a98f0 2024-03-25T18:13:30.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.222+0000 7f8013fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:30.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.222+0000 7f8013fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:30.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.222+0000 7f8013fff640 10 monclient: dump: 2024-03-25T18:13:30.223 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:30.223 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:30.223 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:30.224 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:30.224 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:30.224 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:30.224 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:30.224 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:30.224 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:30.224 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:30.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.222+0000 7f802393d640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:30.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.222+0000 7f802393d640 4 set_mon_vals no callback set 2024-03-25T18:13:30.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.223+0000 7f802393d640 10 monclient: shutdown 2024-03-25T18:13:30.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.223+0000 7f802393d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:30.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.223+0000 7f802393d640 1 -- 172.21.15.63:0/1140117403 >> v1:172.21.15.81:6789/0 conn(0x7f801c0a98f0 legacy=0x7f801c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:30.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.223+0000 7f802393d640 1 -- 172.21.15.63:0/1140117403 shutdown_connections 2024-03-25T18:13:30.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.223+0000 7f802393d640 1 -- 172.21.15.63:0/1140117403 >> 172.21.15.63:0/1140117403 conn(0x7f801c0b7ea0 msgr2=0x7f801c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:30.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.223+0000 7f802393d640 1 -- 172.21.15.63:0/1140117403 shutdown_connections 2024-03-25T18:13:30.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.223+0000 7f802393d640 1 -- 172.21.15.63:0/1140117403 wait complete. 2024-03-25T18:13:30.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.224+0000 7f802393d640 10 monclient: build_initial_monmap 2024-03-25T18:13:30.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.224+0000 7f802393d640 10 monclient: monmap: 2024-03-25T18:13:30.226 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:13:30.226 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:30.226 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:13:30.226 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:13:30.226 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:30.226 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:13:30.226 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:13:30.226 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:13:30.227 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:13:30.227 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:30.227 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.224+0000 7f802393d640 1 librados: starting msgr at 2024-03-25T18:13:30.227 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.224+0000 7f802393d640 1 librados: starting objecter 2024-03-25T18:13:30.227 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.225+0000 7f802393d640 1 Processor -- start 2024-03-25T18:13:30.227 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.225+0000 7f802393d640 1 -- start start 2024-03-25T18:13:30.227 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.225+0000 7f802393d640 1 librados: setting wanted keys 2024-03-25T18:13:30.227 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.225+0000 7f802393d640 1 librados: calling monclient init 2024-03-25T18:13:30.227 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.225+0000 7f802393d640 10 monclient: init 2024-03-25T18:13:30.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.226+0000 7f802393d640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:30.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.226+0000 7f802393d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:30.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.226+0000 7f802393d640 10 monclient(hunting): picked mon.noname-c con 0x7f801c0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:13:30.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.226+0000 7f802393d640 10 monclient(hunting): picked mon.noname-a con 0x7f801c13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:13:30.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.226+0000 7f802393d640 10 monclient(hunting): picked mon.noname-b con 0x7f801c1430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:13:30.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.226+0000 7f802393d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c1390f0 con 0x7f801c13f710 2024-03-25T18:13:30.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.226+0000 7f802393d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c138dd0 con 0x7f801c1430c0 2024-03-25T18:13:30.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.226+0000 7f802393d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c1346b0 con 0x7f801c0a98f0 2024-03-25T18:13:30.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.226+0000 7f802393d640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:30.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.226+0000 7f802393d640 10 monclient(hunting): authenticate will time out at 7504.412748s 2024-03-25T18:13:30.241 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.240+0000 7f8021eda640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f801c1430c0 0x7f801c133fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:39298/0 (socket says 172.21.15.63:39298) 2024-03-25T18:13:30.241 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.240+0000 7f8021eda640 1 -- 172.21.15.63:0/2935961429 learned_addr learned my addr 172.21.15.63:0/2935961429 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:13:30.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.246+0000 7f8011ffb640 10 client.?.objecter ms_handle_connect 0x7f801c13f710 2024-03-25T18:13:30.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.246+0000 7f8011ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:30.248 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.247+0000 7f8011ffb640 10 client.?.objecter ms_handle_connect 0x7f801c1430c0 2024-03-25T18:13:30.248 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.247+0000 7f8011ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:30.249 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.249+0000 7f8011ffb640 10 client.?.objecter ms_handle_connect 0x7f801c0a98f0 2024-03-25T18:13:30.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.249+0000 7f8011ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:30.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.256+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110461953 0 0) 0x7f801c1390f0 con 0x7f801c13f710 2024-03-25T18:13:30.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.256+0000 7f8011ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:30.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.256+0000 7f8011ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:30.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.256+0000 7f8011ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:30.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.256+0000 7f8011ffb640 10 monclient(hunting): my global_id is 4254 2024-03-25T18:13:30.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.256+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8000004eb0 con 0x7f801c13f710 2024-03-25T18:13:30.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.257+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050807140 0 0) 0x7f801c138dd0 con 0x7f801c1430c0 2024-03-25T18:13:30.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.257+0000 7f8011ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:30.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.257+0000 7f8011ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:30.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.257+0000 7f8011ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:30.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.257+0000 7f8011ffb640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:13:30.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.257+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f801c1390f0 con 0x7f801c1430c0 2024-03-25T18:13:30.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.260+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194810186 0 0) 0x7f801c1346b0 con 0x7f801c0a98f0 2024-03-25T18:13:30.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.260+0000 7f8011ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:30.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.260+0000 7f8011ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:30.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.260+0000 7f8011ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:30.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.260+0000 7f8011ffb640 10 monclient(hunting): my global_id is 4265 2024-03-25T18:13:30.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.260+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f801c138dd0 con 0x7f801c0a98f0 2024-03-25T18:13:30.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.267+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1875018427 0 0) 0x7f8000004eb0 con 0x7f801c13f710 2024-03-25T18:13:30.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.267+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f801c1346b0 con 0x7f801c13f710 2024-03-25T18:13:30.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.267+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f800c0018f0 con 0x7f801c13f710 2024-03-25T18:13:30.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.267+0000 7f8011ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:30.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.267+0000 7f8011ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:30.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.267+0000 7f8011ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:30.268 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:30.268 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:30.268 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:30.269 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:30.269 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:30.269 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:30.269 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:30.269 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:30.269 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:30.269 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:30.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.267+0000 7f8011ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:30.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.273+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 442894870 0 0) 0x7f801c1390f0 con 0x7f801c1430c0 2024-03-25T18:13:30.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.273+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8000004eb0 con 0x7f801c1430c0 2024-03-25T18:13:30.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.273+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8014001b30 con 0x7f801c1430c0 2024-03-25T18:13:30.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.273+0000 7f8011ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:30.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.273+0000 7f8011ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:30.275 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.273+0000 7f8011ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:30.275 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:30.275 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:30.275 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:30.275 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:30.275 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:30.275 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:30.275 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:30.275 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:30.275 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:30.275 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:30.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3172468611 0 0) 0x7f801c1346b0 con 0x7f801c13f710 2024-03-25T18:13:30.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 >> v1:172.21.15.63:6790/0 conn(0x7f801c0a98f0 legacy=0x7f801c137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:30.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 >> v1:172.21.15.81:6789/0 conn(0x7f801c1430c0 legacy=0x7f801c133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:30.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f8011ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:30.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f8011ffb640 1 monclient: found mon.a 2024-03-25T18:13:30.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f8011ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:30.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f8011ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:30.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801c134840 con 0x7f801c13f710 2024-03-25T18:13:30.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f8011ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:30.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f8011ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:30.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f8011ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:30.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f802393d640 5 monclient: authenticate success, global_id 4254 2024-03-25T18:13:30.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f802393d640 10 monclient: _renew_subs 2024-03-25T18:13:30.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f802393d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:30.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f802393d640 1 -- 172.21.15.63:0/2935961429 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801c134a90 con 0x7f801c13f710 2024-03-25T18:13:30.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f802393d640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:30.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f802393d640 10 monclient: _renew_subs 2024-03-25T18:13:30.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f802393d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:30.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f802393d640 1 -- 172.21.15.63:0/2935961429 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f801c1356f0 con 0x7f801c13f710 2024-03-25T18:13:30.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.276+0000 7f802393d640 1 librados: init done 2024-03-25T18:13:30.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.277+0000 7f802393d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:30.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.277+0000 7f802393d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:30.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.277+0000 7f802393d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:30.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.277+0000 7f802393d640 1 -- 172.21.15.63:0/2935961429 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801c0627b0 con 0x7f801c13f710 2024-03-25T18:13:30.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.589+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f800c003610 con 0x7f801c13f710 2024-03-25T18:13:30.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.589+0000 7f8011ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:30.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.589+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f800c003920 con 0x7f801c13f710 2024-03-25T18:13:30.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.589+0000 7f8011ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:30.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.589+0000 7f8011ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:30.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.589+0000 7f8011ffb640 10 monclient: dump: 2024-03-25T18:13:30.590 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:30.590 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:30.590 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:30.590 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:30.590 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:30.591 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:30.591 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:30.591 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:30.591 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:30.591 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:30.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.589+0000 7f80137fe640 4 set_mon_vals no callback set 2024-03-25T18:13:30.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.590+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f800c0198f0 con 0x7f801c13f710 2024-03-25T18:13:30.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.590+0000 7f80216d9640 10 monclient: get_auth_request con 0x7f800006df70 auth_method 0 2024-03-25T18:13:30.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.591+0000 7f8011ffb640 10 client.4254.objecter ms_handle_connect 0x7f800006df70 2024-03-25T18:13:30.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.595+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f800c0535e0 con 0x7f801c13f710 2024-03-25T18:13:30.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.595+0000 7f8011ffb640 10 client.4254.objecter ms_dispatch 0x7f801c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:30.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.595+0000 7f8011ffb640 3 client.4254.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:30.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.595+0000 7f8011ffb640 3 client.4254.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:30.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.595+0000 7f8011ffb640 20 client.4254.objecter dump_active .. 0 homeless 2024-03-25T18:13:30.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.595+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f800c04ef20 con 0x7f801c13f710 2024-03-25T18:13:30.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.595+0000 7f8011ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:30.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.595+0000 7f8011ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:30.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.808+0000 7f802393d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:30.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.808+0000 7f802393d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:30.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.808+0000 7f802393d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:30.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.808+0000 7f802393d640 1 -- 172.21.15.63:0/2935961429 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f801c1361b0 con 0x7f801c13f710 2024-03-25T18:13:30.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.849+0000 7f8011ffb640 1 -- 172.21.15.63:0/2935961429 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f800c051020 con 0x7f801c13f710 2024-03-25T18:13:30.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.849+0000 7f8011ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:30.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.849+0000 7f8011ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:30.851 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:13:30.851 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:30.853 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.852+0000 7f802393d640 10 librados: watch_flush enter 2024-03-25T18:13:30.853 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.852+0000 7f802393d640 10 librados: watch_flush exit 2024-03-25T18:13:30.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.852+0000 7f802393d640 20 client.4254.objecter shutdown clearing up homeless session... 2024-03-25T18:13:30.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.852+0000 7f802393d640 10 client.4254.objecter successfully canceled tick 2024-03-25T18:13:30.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.852+0000 7f802393d640 1 -- 172.21.15.63:0/2935961429 >> v1:172.21.15.81:6800/3640340873 conn(0x7f800006df70 legacy=0x7f8000070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:30.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.852+0000 7f802393d640 10 monclient: shutdown 2024-03-25T18:13:30.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.852+0000 7f802393d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:30.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.852+0000 7f802393d640 1 -- 172.21.15.63:0/2935961429 >> v1:172.21.15.63:6789/0 conn(0x7f801c13f710 legacy=0x7f801c1386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:30.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.852+0000 7f802393d640 1 -- 172.21.15.63:0/2935961429 shutdown_connections 2024-03-25T18:13:30.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.852+0000 7f802393d640 1 -- 172.21.15.63:0/2935961429 >> 172.21.15.63:0/2935961429 conn(0x7f801c0b7ea0 msgr2=0x7f801c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:30.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.852+0000 7f802393d640 1 -- 172.21.15.63:0/2935961429 shutdown_connections 2024-03-25T18:13:30.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.852+0000 7f802393d640 1 -- 172.21.15.63:0/2935961429 wait complete. 2024-03-25T18:13:30.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:30.852+0000 7f802393d640 1 librados: shutdown 2024-03-25T18:13:30.865 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:33.154 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:01:14.846 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:33.221 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:01:14.914 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:33.251 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:01:14.912 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:33.324 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:01:14.985 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:33.357 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:01:15.050 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:33.425 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:01:15.118 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:33.470 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:01:15.162 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:33.542 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:01:15.202 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:33.676 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:01:15.368 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:37.672 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:37.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.787+0000 7fc3d3f8f640 10 monclient: get_monmap_and_config 2024-03-25T18:13:37.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.788+0000 7fc3d3f8f640 10 monclient: build_initial_monmap 2024-03-25T18:13:37.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.788+0000 7fc3d3f8f640 10 monclient: monmap: 2024-03-25T18:13:37.789 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:13:37.789 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:37.789 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:13:37.789158+0000 2024-03-25T18:13:37.789 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:13:37.789158+0000 2024-03-25T18:13:37.789 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:37.789 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:37.789 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:13:37.789 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:13:37.789 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:13:37.789 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:37.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.788+0000 7fc3d3f8f640 1 Processor -- start 2024-03-25T18:13:37.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.789+0000 7fc3d3f8f640 1 -- start start 2024-03-25T18:13:37.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.789+0000 7fc3d3f8f640 10 monclient: init 2024-03-25T18:13:37.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.789+0000 7fc3d3f8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:37.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.789+0000 7fc3d3f8f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:37.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.789+0000 7fc3d3f8f640 10 monclient(hunting): picked mon.noname-c con 0x7fc3cc0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:13:37.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.789+0000 7fc3d3f8f640 10 monclient(hunting): picked mon.noname-a con 0x7fc3cc13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:13:37.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.789+0000 7fc3d3f8f640 10 monclient(hunting): picked mon.noname-b con 0x7fc3cc1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:13:37.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.789+0000 7fc3d3f8f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3cc142520 con 0x7fc3cc13f710 2024-03-25T18:13:37.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.789+0000 7fc3d3f8f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3cc0b28a0 con 0x7fc3cc0a98f0 2024-03-25T18:13:37.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.789+0000 7fc3d3f8f640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3cc0b2a30 con 0x7fc3cc1430c0 2024-03-25T18:13:37.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.789+0000 7fc3d3f8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:37.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.789+0000 7fc3d3f8f640 10 monclient(hunting): authenticate will time out at 7511.976047s 2024-03-25T18:13:37.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.797+0000 7fc3d152a640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fc3cc13f710 0x7fc3cc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:58226/0 (socket says 172.21.15.63:58226) 2024-03-25T18:13:37.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.797+0000 7fc3d152a640 1 -- 172.21.15.63:0/3757212098 learned_addr learned my addr 172.21.15.63:0/3757212098 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:13:37.814 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.814+0000 7fc3d0d29640 1 -- 172.21.15.63:0/3757212098 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145788950 0 0) 0x7fc3cc142520 con 0x7fc3cc13f710 2024-03-25T18:13:37.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.814+0000 7fc3d0d29640 10 monclient(hunting): _negotiate 2024-03-25T18:13:37.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.814+0000 7fc3d0d29640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:37.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.814+0000 7fc3d0d29640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:37.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.814+0000 7fc3d0d29640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:13:37.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.814+0000 7fc3d0d29640 1 -- 172.21.15.63:0/3757212098 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3b8003520 con 0x7fc3cc13f710 2024-03-25T18:13:37.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.821+0000 7fc3d0d29640 1 -- 172.21.15.63:0/3757212098 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263999605 0 0) 0x7fc3cc0b28a0 con 0x7fc3cc0a98f0 2024-03-25T18:13:37.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.821+0000 7fc3d0d29640 10 monclient(hunting): _negotiate 2024-03-25T18:13:37.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.821+0000 7fc3d0d29640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:37.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.821+0000 7fc3d0d29640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:37.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.821+0000 7fc3d0d29640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:13:37.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.821+0000 7fc3d0d29640 1 -- 172.21.15.63:0/3757212098 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3cc142520 con 0x7fc3cc0a98f0 2024-03-25T18:13:37.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.824+0000 7fc3d0d29640 1 -- 172.21.15.63:0/3757212098 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066657576 0 0) 0x7fc3cc0b2a30 con 0x7fc3cc1430c0 2024-03-25T18:13:37.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.824+0000 7fc3d0d29640 10 monclient(hunting): _negotiate 2024-03-25T18:13:37.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.824+0000 7fc3d0d29640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:37.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.824+0000 7fc3d0d29640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:37.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.824+0000 7fc3d0d29640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:13:37.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.824+0000 7fc3d0d29640 1 -- 172.21.15.63:0/3757212098 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3cc0b28a0 con 0x7fc3cc1430c0 2024-03-25T18:13:37.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d0d29640 1 -- 172.21.15.63:0/3757212098 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1951441538 0 0) 0x7fc3b8003520 con 0x7fc3cc13f710 2024-03-25T18:13:37.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d0d29640 1 -- 172.21.15.63:0/3757212098 >> v1:172.21.15.63:6790/0 conn(0x7fc3cc1430c0 legacy=0x7fc3cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:37.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d0d29640 1 -- 172.21.15.63:0/3757212098 >> v1:172.21.15.81:6789/0 conn(0x7fc3cc0a98f0 legacy=0x7fc3cc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:37.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d0d29640 10 monclient: _finish_hunting 0 2024-03-25T18:13:37.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d0d29640 1 monclient: found mon.noname-a 2024-03-25T18:13:37.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d0d29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:37.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d0d29640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:13:37.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d0d29640 1 -- 172.21.15.63:0/3757212098 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3cc0b44c0 con 0x7fc3cc13f710 2024-03-25T18:13:37.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d0d29640 10 monclient: _finish_auth 0 2024-03-25T18:13:37.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d0d29640 10 monclient: _check_auth_tickets 2024-03-25T18:13:37.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d0d29640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:37.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d0d29640 1 -- 172.21.15.63:0/3757212098 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc3bc001690 con 0x7fc3cc13f710 2024-03-25T18:13:37.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d0d29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:37.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d0d29640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:37.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d0d29640 10 monclient: dump: 2024-03-25T18:13:37.827 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:37.827 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:37.827 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:37.828 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:37.828 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:37.828 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:37.828 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:37.828 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:37.828 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:37.828 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:37.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d3f8f640 5 monclient: authenticate success, global_id 4257 2024-03-25T18:13:37.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.825+0000 7fc3d3f8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:37.837 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.837+0000 7fc3d0d29640 1 -- 172.21.15.63:0/3757212098 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3bc0017f0 con 0x7fc3cc13f710 2024-03-25T18:13:37.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.837+0000 7fc3d0d29640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:37.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.837+0000 7fc3d0d29640 1 -- 172.21.15.63:0/3757212098 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc3bc001c00 con 0x7fc3cc13f710 2024-03-25T18:13:37.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.837+0000 7fc3d0d29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:37.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.837+0000 7fc3d0d29640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:37.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.837+0000 7fc3d0d29640 10 monclient: dump: 2024-03-25T18:13:37.838 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:37.838 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:37.838 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:37.838 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:37.838 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:37.838 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:37.838 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:37.839 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:37.839 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:37.839 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:37.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.837+0000 7fc3d3f8f640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:37.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.837+0000 7fc3d3f8f640 4 set_mon_vals no callback set 2024-03-25T18:13:37.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.838+0000 7fc3d3f8f640 10 monclient: shutdown 2024-03-25T18:13:37.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.838+0000 7fc3d3f8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:37.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.838+0000 7fc3d3f8f640 1 -- 172.21.15.63:0/3757212098 >> v1:172.21.15.63:6789/0 conn(0x7fc3cc13f710 legacy=0x7fc3cc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:37.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.838+0000 7fc3d3f8f640 1 -- 172.21.15.63:0/3757212098 shutdown_connections 2024-03-25T18:13:37.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.838+0000 7fc3d3f8f640 1 -- 172.21.15.63:0/3757212098 >> 172.21.15.63:0/3757212098 conn(0x7fc3cc0b7ea0 msgr2=0x7fc3cc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:37.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.838+0000 7fc3d3f8f640 1 -- 172.21.15.63:0/3757212098 shutdown_connections 2024-03-25T18:13:37.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.838+0000 7fc3d3f8f640 1 -- 172.21.15.63:0/3757212098 wait complete. 2024-03-25T18:13:37.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.839+0000 7fc3d3f8f640 10 monclient: build_initial_monmap 2024-03-25T18:13:37.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.839+0000 7fc3d3f8f640 10 monclient: monmap: 2024-03-25T18:13:37.841 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:13:37.841 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:37.841 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:13:37.841 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:13:37.841 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:37.841 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:13:37.841 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:13:37.841 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:13:37.841 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:13:37.841 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:37.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.839+0000 7fc3d3f8f640 1 librados: starting msgr at 2024-03-25T18:13:37.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.839+0000 7fc3d3f8f640 1 librados: starting objecter 2024-03-25T18:13:37.842 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.839+0000 7fc3d3f8f640 1 Processor -- start 2024-03-25T18:13:37.842 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.839+0000 7fc3d3f8f640 1 -- start start 2024-03-25T18:13:37.842 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.839+0000 7fc3d3f8f640 1 librados: setting wanted keys 2024-03-25T18:13:37.842 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.839+0000 7fc3d3f8f640 1 librados: calling monclient init 2024-03-25T18:13:37.842 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.839+0000 7fc3d3f8f640 10 monclient: init 2024-03-25T18:13:37.843 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.840+0000 7fc3d3f8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:37.843 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.840+0000 7fc3d3f8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:37.843 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.840+0000 7fc3d3f8f640 10 monclient(hunting): picked mon.noname-a con 0x7fc3cc0a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:13:37.843 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.840+0000 7fc3d3f8f640 10 monclient(hunting): picked mon.noname-c con 0x7fc3cc13f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:13:37.843 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.840+0000 7fc3d3f8f640 10 monclient(hunting): picked mon.noname-b con 0x7fc3cc1430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:13:37.843 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.840+0000 7fc3d3f8f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3cc0abac0 con 0x7fc3cc0a98f0 2024-03-25T18:13:37.843 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.840+0000 7fc3d3f8f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3cc0abda0 con 0x7fc3cc1430c0 2024-03-25T18:13:37.843 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.840+0000 7fc3d3f8f640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3cc1380e0 con 0x7fc3cc13f710 2024-03-25T18:13:37.843 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.840+0000 7fc3d3f8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:37.843 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.840+0000 7fc3d3f8f640 10 monclient(hunting): authenticate will time out at 7512.026909s 2024-03-25T18:13:37.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.849+0000 7fc3d252c640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc3cc1430c0 0x7fc3cc1e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:43600/0 (socket says 172.21.15.63:43600) 2024-03-25T18:13:37.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.849+0000 7fc3d252c640 1 -- 172.21.15.63:0/3955547778 learned_addr learned my addr 172.21.15.63:0/3955547778 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:13:37.855 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.854+0000 7fc3c27fc640 10 client.?.objecter ms_handle_connect 0x7fc3cc1430c0 2024-03-25T18:13:37.855 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.854+0000 7fc3c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.863+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1612462038 0 0) 0x7fc3cc0abda0 con 0x7fc3cc1430c0 2024-03-25T18:13:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.863+0000 7fc3c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.863+0000 7fc3c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.863+0000 7fc3c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.863+0000 7fc3c27fc640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:13:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.863+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3b0004eb0 con 0x7fc3cc1430c0 2024-03-25T18:13:37.868 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.868+0000 7fc3c27fc640 10 client.?.objecter ms_handle_connect 0x7fc3cc13f710 2024-03-25T18:13:37.869 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.868+0000 7fc3c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:37.870 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.869+0000 7fc3c27fc640 10 client.?.objecter ms_handle_connect 0x7fc3cc0a98f0 2024-03-25T18:13:37.870 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.869+0000 7fc3c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:37.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.875+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 957896615 0 0) 0x7fc3b0004eb0 con 0x7fc3cc1430c0 2024-03-25T18:13:37.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.875+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3cc0abda0 con 0x7fc3cc1430c0 2024-03-25T18:13:37.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.877+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964962636 0 0) 0x7fc3cc0abac0 con 0x7fc3cc0a98f0 2024-03-25T18:13:37.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.877+0000 7fc3c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:37.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.877+0000 7fc3c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:37.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.877+0000 7fc3c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:37.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.877+0000 7fc3c27fc640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:13:37.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.877+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3b0004eb0 con 0x7fc3cc0a98f0 2024-03-25T18:13:37.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.878+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81770806 0 0) 0x7fc3cc1380e0 con 0x7fc3cc13f710 2024-03-25T18:13:37.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.878+0000 7fc3c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:37.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.878+0000 7fc3c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:37.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.878+0000 7fc3c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:37.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.878+0000 7fc3c27fc640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:13:37.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.878+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3cc0abac0 con 0x7fc3cc13f710 2024-03-25T18:13:37.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.880+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc3c40019f0 con 0x7fc3cc1430c0 2024-03-25T18:13:37.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.880+0000 7fc3c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:37.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.880+0000 7fc3c27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:37.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.880+0000 7fc3c27fc640 10 monclient(hunting): dump: 2024-03-25T18:13:37.881 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:37.881 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:37.881 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:37.881 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:37.881 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:37.881 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:37.881 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:37.881 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:37.881 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:37.882 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:37.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.880+0000 7fc3c27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:37.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.888+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4069284071 0 0) 0x7fc3b0004eb0 con 0x7fc3cc0a98f0 2024-03-25T18:13:37.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.888+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3cc1380e0 con 0x7fc3cc0a98f0 2024-03-25T18:13:37.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.888+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc3c8001cb0 con 0x7fc3cc0a98f0 2024-03-25T18:13:37.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.888+0000 7fc3c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:37.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.888+0000 7fc3c27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:37.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.888+0000 7fc3c27fc640 10 monclient(hunting): dump: 2024-03-25T18:13:37.889 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:37.889 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:37.889 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:37.890 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:37.890 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:37.890 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:37.890 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:37.890 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:37.890 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:37.890 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:37.893 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.892+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2253482583 0 0) 0x7fc3cc0abac0 con 0x7fc3cc13f710 2024-03-25T18:13:37.893 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.892+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3b0004eb0 con 0x7fc3cc13f710 2024-03-25T18:13:37.893 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.892+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc3bc001b00 con 0x7fc3cc13f710 2024-03-25T18:13:37.893 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.892+0000 7fc3c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:37.893 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.892+0000 7fc3c27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:37.893 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.892+0000 7fc3c27fc640 10 monclient(hunting): dump: 2024-03-25T18:13:37.893 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:37.893 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:37.893 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:37.894 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:37.894 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:37.894 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:37.894 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:37.894 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:37.894 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:37.894 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:37.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1629898445 0 0) 0x7fc3cc0abda0 con 0x7fc3cc1430c0 2024-03-25T18:13:37.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 >> v1:172.21.15.63:6790/0 conn(0x7fc3cc13f710 legacy=0x7fc3cc138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:37.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 >> v1:172.21.15.63:6789/0 conn(0x7fc3cc0a98f0 legacy=0x7fc3cc0ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:37.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3c27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:37.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3c27fc640 1 monclient: found mon.b 2024-03-25T18:13:37.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:37.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3c27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:13:37.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3cc137d90 con 0x7fc3cc1430c0 2024-03-25T18:13:37.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3c27fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:37.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3c27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:37.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:37.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3d3f8f640 5 monclient: authenticate success, global_id 4270 2024-03-25T18:13:37.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3d3f8f640 10 monclient: _renew_subs 2024-03-25T18:13:37.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3d3f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:13:37.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3d3f8f640 1 -- 172.21.15.63:0/3955547778 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3cc139240 con 0x7fc3cc1430c0 2024-03-25T18:13:37.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3d3f8f640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:37.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3d3f8f640 10 monclient: _renew_subs 2024-03-25T18:13:37.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3d3f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:13:37.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3d3f8f640 1 -- 172.21.15.63:0/3955547778 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3cc136830 con 0x7fc3cc1430c0 2024-03-25T18:13:37.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.896+0000 7fc3d3f8f640 1 librados: init done 2024-03-25T18:13:37.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.897+0000 7fc3d3f8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:37.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.897+0000 7fc3d3f8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:37.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.897+0000 7fc3d3f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:13:37.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:37.897+0000 7fc3d3f8f640 1 -- 172.21.15.63:0/3955547778 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3cc0627b0 con 0x7fc3cc1430c0 2024-03-25T18:13:38.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.185+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3c40035f0 con 0x7fc3cc1430c0 2024-03-25T18:13:38.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.185+0000 7fc3c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:38.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.185+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc3c4003a00 con 0x7fc3cc1430c0 2024-03-25T18:13:38.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.185+0000 7fc3c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:38.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.185+0000 7fc3c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:38.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.185+0000 7fc3c27fc640 10 monclient: dump: 2024-03-25T18:13:38.187 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:38.187 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:38.187 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:38.187 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:38.187 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:38.187 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:38.187 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:38.187 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:38.187 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:38.187 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:38.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.185+0000 7fc3c3fff640 4 set_mon_vals no callback set 2024-03-25T18:13:38.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.185+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fc3c4003c20 con 0x7fc3cc1430c0 2024-03-25T18:13:38.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.186+0000 7fc3d1d2b640 10 monclient: get_auth_request con 0x7fc3b006e180 auth_method 0 2024-03-25T18:13:38.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.186+0000 7fc3c27fc640 10 client.4270.objecter ms_handle_connect 0x7fc3b006e180 2024-03-25T18:13:38.191 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.190+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fc3c4054760 con 0x7fc3cc1430c0 2024-03-25T18:13:38.191 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.190+0000 7fc3c27fc640 10 client.4270.objecter ms_dispatch 0x7fc3cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:38.191 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.190+0000 7fc3c27fc640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:38.191 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.190+0000 7fc3c27fc640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:38.191 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.190+0000 7fc3c27fc640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T18:13:38.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.191+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3c4050270 con 0x7fc3cc1430c0 2024-03-25T18:13:38.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.191+0000 7fc3c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:38.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.191+0000 7fc3c27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:38.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.450+0000 7fc3d3f8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:38.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.450+0000 7fc3d3f8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:38.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.450+0000 7fc3d3f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:13:38.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.450+0000 7fc3d3f8f640 1 -- 172.21.15.63:0/3955547778 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3cc13f070 con 0x7fc3cc1430c0 2024-03-25T18:13:38.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.488+0000 7fc3c27fc640 1 -- 172.21.15.63:0/3955547778 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fc3c40521a0 con 0x7fc3cc1430c0 2024-03-25T18:13:38.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.488+0000 7fc3c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:38.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.488+0000 7fc3c27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:38.490 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:13:38.490 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:38.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.491+0000 7fc3d3f8f640 10 librados: watch_flush enter 2024-03-25T18:13:38.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.491+0000 7fc3d3f8f640 10 librados: watch_flush exit 2024-03-25T18:13:38.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.491+0000 7fc3d3f8f640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T18:13:38.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.491+0000 7fc3d3f8f640 10 client.4270.objecter successfully canceled tick 2024-03-25T18:13:38.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.491+0000 7fc3d3f8f640 1 -- 172.21.15.63:0/3955547778 >> v1:172.21.15.81:6800/3640340873 conn(0x7fc3b006e180 legacy=0x7fc3b0070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:38.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.491+0000 7fc3d3f8f640 10 monclient: shutdown 2024-03-25T18:13:38.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.491+0000 7fc3d3f8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:38.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.491+0000 7fc3d3f8f640 1 -- 172.21.15.63:0/3955547778 >> v1:172.21.15.81:6789/0 conn(0x7fc3cc1430c0 legacy=0x7fc3cc1e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:38.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.491+0000 7fc3d3f8f640 1 -- 172.21.15.63:0/3955547778 shutdown_connections 2024-03-25T18:13:38.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.491+0000 7fc3d3f8f640 1 -- 172.21.15.63:0/3955547778 >> 172.21.15.63:0/3955547778 conn(0x7fc3cc0b7ea0 msgr2=0x7fc3cc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:38.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.491+0000 7fc3d3f8f640 1 -- 172.21.15.63:0/3955547778 shutdown_connections 2024-03-25T18:13:38.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.491+0000 7fc3d3f8f640 1 -- 172.21.15.63:0/3955547778 wait complete. 2024-03-25T18:13:38.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:38.491+0000 7fc3d3f8f640 1 librados: shutdown 2024-03-25T18:13:38.504 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:45.311 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:45.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.411+0000 7fea3e73d640 10 monclient: get_monmap_and_config 2024-03-25T18:13:45.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.411+0000 7fea3e73d640 10 monclient: build_initial_monmap 2024-03-25T18:13:45.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.411+0000 7fea3e73d640 10 monclient: monmap: 2024-03-25T18:13:45.412 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:13:45.412 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:45.412 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:13:45.412201+0000 2024-03-25T18:13:45.412 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:13:45.412201+0000 2024-03-25T18:13:45.412 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:45.412 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:45.413 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:13:45.413 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:13:45.413 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:13:45.413 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:45.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.411+0000 7fea3e73d640 1 Processor -- start 2024-03-25T18:13:45.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.411+0000 7fea3e73d640 1 -- start start 2024-03-25T18:13:45.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.411+0000 7fea3e73d640 10 monclient: init 2024-03-25T18:13:45.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.412+0000 7fea3e73d640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:45.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.412+0000 7fea3e73d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:45.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.412+0000 7fea3e73d640 10 monclient(hunting): picked mon.noname-b con 0x7fea380b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T18:13:45.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.412+0000 7fea3e73d640 10 monclient(hunting): picked mon.noname-c con 0x7fea380b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:13:45.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.412+0000 7fea3e73d640 10 monclient(hunting): picked mon.noname-a con 0x7fea3814be90 addr v1:172.21.15.63:6789/0 2024-03-25T18:13:45.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.412+0000 7fea3e73d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea380af940 con 0x7fea3814be90 2024-03-25T18:13:45.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.412+0000 7fea3e73d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea380b0a70 con 0x7fea380b1730 2024-03-25T18:13:45.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.412+0000 7fea3e73d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea380b0d50 con 0x7fea380b4d60 2024-03-25T18:13:45.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.412+0000 7fea3e73d640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:45.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.412+0000 7fea3e73d640 10 monclient(hunting): authenticate will time out at 7519.598914s 2024-03-25T18:13:45.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.413+0000 7fea377fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fea380b1730 0x7fea380b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:36216/0 (socket says 172.21.15.63:36216) 2024-03-25T18:13:45.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.413+0000 7fea377fe640 1 -- 172.21.15.63:0/3910474993 learned_addr learned my addr 172.21.15.63:0/3910474993 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:13:45.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.441+0000 7fea36ffd640 1 -- 172.21.15.63:0/3910474993 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878976699 0 0) 0x7fea380af940 con 0x7fea3814be90 2024-03-25T18:13:45.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.441+0000 7fea36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:45.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.441+0000 7fea36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:45.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.441+0000 7fea36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:45.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.441+0000 7fea36ffd640 10 monclient(hunting): my global_id is 4263 2024-03-25T18:13:45.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.441+0000 7fea36ffd640 1 -- 172.21.15.63:0/3910474993 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea20003520 con 0x7fea3814be90 2024-03-25T18:13:45.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea36ffd640 1 -- 172.21.15.63:0/3910474993 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2006897117 0 0) 0x7fea20003520 con 0x7fea3814be90 2024-03-25T18:13:45.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea36ffd640 1 -- 172.21.15.63:0/3910474993 >> v1:172.21.15.63:6790/0 conn(0x7fea380b4d60 legacy=0x7fea3813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:45.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea36ffd640 1 -- 172.21.15.63:0/3910474993 >> v1:172.21.15.81:6789/0 conn(0x7fea380b1730 legacy=0x7fea380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:45.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea36ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:13:45.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea36ffd640 1 monclient: found mon.noname-a 2024-03-25T18:13:45.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:45.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea36ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:13:45.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea36ffd640 1 -- 172.21.15.63:0/3910474993 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea380b1030 con 0x7fea3814be90 2024-03-25T18:13:45.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea36ffd640 10 monclient: _finish_auth 0 2024-03-25T18:13:45.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea36ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:13:45.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:45.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea36ffd640 1 -- 172.21.15.63:0/3910474993 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fea30001a10 con 0x7fea3814be90 2024-03-25T18:13:45.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea3e73d640 5 monclient: authenticate success, global_id 4263 2024-03-25T18:13:45.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea3e73d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:45.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:45.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:45.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea36ffd640 10 monclient: dump: 2024-03-25T18:13:45.459 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:45.459 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:45.459 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:45.459 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:45.459 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:45.459 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:45.459 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:45.459 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:45.459 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:45.460 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:45.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.457+0000 7fea3e73d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:45.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.466+0000 7fea36ffd640 1 -- 172.21.15.63:0/3910474993 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea30001b70 con 0x7fea3814be90 2024-03-25T18:13:45.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.466+0000 7fea36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:45.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.466+0000 7fea36ffd640 1 -- 172.21.15.63:0/3910474993 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fea30001f80 con 0x7fea3814be90 2024-03-25T18:13:45.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.466+0000 7fea36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:45.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.466+0000 7fea36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:45.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.466+0000 7fea36ffd640 10 monclient: dump: 2024-03-25T18:13:45.467 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:45.467 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:45.467 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:45.467 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:45.467 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:45.468 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:45.468 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:45.468 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:45.468 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:45.468 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:45.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.466+0000 7fea3e73d640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:45.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.466+0000 7fea3e73d640 4 set_mon_vals no callback set 2024-03-25T18:13:45.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.467+0000 7fea3e73d640 10 monclient: shutdown 2024-03-25T18:13:45.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.467+0000 7fea3e73d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:45.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.467+0000 7fea3e73d640 1 -- 172.21.15.63:0/3910474993 >> v1:172.21.15.63:6789/0 conn(0x7fea3814be90 legacy=0x7fea3814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:45.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.467+0000 7fea3e73d640 1 -- 172.21.15.63:0/3910474993 shutdown_connections 2024-03-25T18:13:45.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.467+0000 7fea3e73d640 1 -- 172.21.15.63:0/3910474993 >> 172.21.15.63:0/3910474993 conn(0x7fea380617a0 msgr2=0x7fea38061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:45.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.467+0000 7fea3e73d640 1 -- 172.21.15.63:0/3910474993 shutdown_connections 2024-03-25T18:13:45.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.467+0000 7fea3e73d640 1 -- 172.21.15.63:0/3910474993 wait complete. 2024-03-25T18:13:45.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.468+0000 7fea3e73d640 10 monclient: build_initial_monmap 2024-03-25T18:13:45.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.468+0000 7fea3e73d640 10 monclient: monmap: 2024-03-25T18:13:45.470 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:13:45.470 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:45.470 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:13:45.470 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:13:45.470 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:45.471 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:13:45.471 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:13:45.471 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:13:45.471 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:13:45.471 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:45.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.468+0000 7fea3e73d640 1 librados: starting msgr at 2024-03-25T18:13:45.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.468+0000 7fea3e73d640 1 librados: starting objecter 2024-03-25T18:13:45.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.468+0000 7fea3e73d640 1 Processor -- start 2024-03-25T18:13:45.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.469+0000 7fea3e73d640 1 -- start start 2024-03-25T18:13:45.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.469+0000 7fea3e73d640 1 librados: setting wanted keys 2024-03-25T18:13:45.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.469+0000 7fea3e73d640 1 librados: calling monclient init 2024-03-25T18:13:45.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.469+0000 7fea3e73d640 10 monclient: init 2024-03-25T18:13:45.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.469+0000 7fea3e73d640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:45.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.469+0000 7fea3e73d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:45.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.469+0000 7fea3e73d640 10 monclient(hunting): picked mon.noname-c con 0x7fea380b1730 addr v1:172.21.15.63:6790/0 2024-03-25T18:13:45.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.469+0000 7fea3e73d640 10 monclient(hunting): picked mon.noname-b con 0x7fea380b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T18:13:45.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.469+0000 7fea3e73d640 10 monclient(hunting): picked mon.noname-a con 0x7fea3814be90 addr v1:172.21.15.63:6789/0 2024-03-25T18:13:45.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.469+0000 7fea3e73d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea381307c0 con 0x7fea3814be90 2024-03-25T18:13:45.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.469+0000 7fea3e73d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea38138f90 con 0x7fea380b4d60 2024-03-25T18:13:45.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.469+0000 7fea3e73d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea38132ab0 con 0x7fea380b1730 2024-03-25T18:13:45.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.469+0000 7fea3e73d640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:45.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.469+0000 7fea3e73d640 10 monclient(hunting): authenticate will time out at 7519.656059s 2024-03-25T18:13:45.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.473+0000 7fea3ccda640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fea3814be90 0x7fea381300b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:34540/0 (socket says 172.21.15.63:34540) 2024-03-25T18:13:45.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.473+0000 7fea3ccda640 1 -- 172.21.15.63:0/4244593198 learned_addr learned my addr 172.21.15.63:0/4244593198 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:13:45.475 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.474+0000 7fea34ff9640 10 client.?.objecter ms_handle_connect 0x7fea3814be90 2024-03-25T18:13:45.475 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.475+0000 7fea34ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:45.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.483+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363307524 0 0) 0x7fea381307c0 con 0x7fea3814be90 2024-03-25T18:13:45.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.483+0000 7fea34ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:45.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.483+0000 7fea34ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:45.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.483+0000 7fea34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:45.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.483+0000 7fea34ff9640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:13:45.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.483+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea24004eb0 con 0x7fea3814be90 2024-03-25T18:13:45.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.492+0000 7fea34ff9640 10 client.?.objecter ms_handle_connect 0x7fea380b4d60 2024-03-25T18:13:45.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.492+0000 7fea34ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:45.500 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.500+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825005303 0 0) 0x7fea38138f90 con 0x7fea380b4d60 2024-03-25T18:13:45.500 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.500+0000 7fea34ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:45.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.500+0000 7fea34ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:45.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.500+0000 7fea34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:45.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.500+0000 7fea34ff9640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:13:45.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.500+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea381307c0 con 0x7fea380b4d60 2024-03-25T18:13:45.512 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.512+0000 7fea34ff9640 10 client.?.objecter ms_handle_connect 0x7fea380b1730 2024-03-25T18:13:45.512 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.512+0000 7fea34ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:45.520 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.520+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2764761891 0 0) 0x7fea381307c0 con 0x7fea380b4d60 2024-03-25T18:13:45.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.520+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea38138f90 con 0x7fea380b4d60 2024-03-25T18:13:45.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.520+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fea2c0018a0 con 0x7fea380b4d60 2024-03-25T18:13:45.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.520+0000 7fea34ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:45.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.520+0000 7fea34ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:45.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.520+0000 7fea34ff9640 10 monclient(hunting): dump: 2024-03-25T18:13:45.521 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:45.521 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:45.521 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:45.521 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:45.521 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:45.521 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:45.521 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:45.521 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:45.521 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:45.522 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:45.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.520+0000 7fea34ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:45.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.521+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1545364290 0 0) 0x7fea24004eb0 con 0x7fea3814be90 2024-03-25T18:13:45.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.521+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea381307c0 con 0x7fea3814be90 2024-03-25T18:13:45.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.521+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fea30001fc0 con 0x7fea3814be90 2024-03-25T18:13:45.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.521+0000 7fea34ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:45.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.521+0000 7fea34ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:45.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.521+0000 7fea34ff9640 10 monclient(hunting): dump: 2024-03-25T18:13:45.523 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:45.523 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:45.523 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:45.523 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:45.523 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:45.523 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:45.523 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:45.523 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:45.523 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:45.523 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:45.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2766108853 0 0) 0x7fea38138f90 con 0x7fea380b4d60 2024-03-25T18:13:45.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 >> v1:172.21.15.63:6790/0 conn(0x7fea380b1730 legacy=0x7fea381393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:45.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 >> v1:172.21.15.63:6789/0 conn(0x7fea3814be90 legacy=0x7fea381300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:45.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea34ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:13:45.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea34ff9640 1 monclient: found mon.b 2024-03-25T18:13:45.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea34ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:45.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea34ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:13:45.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea38133a40 con 0x7fea380b4d60 2024-03-25T18:13:45.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea34ff9640 10 monclient: _finish_auth 0 2024-03-25T18:13:45.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea34ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:13:45.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:45.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea3e73d640 5 monclient: authenticate success, global_id 4276 2024-03-25T18:13:45.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea3e73d640 10 monclient: _renew_subs 2024-03-25T18:13:45.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea3e73d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:13:45.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea3e73d640 1 -- 172.21.15.63:0/4244593198 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea381316c0 con 0x7fea380b4d60 2024-03-25T18:13:45.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea3e73d640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:45.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea3e73d640 10 monclient: _renew_subs 2024-03-25T18:13:45.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea3e73d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:13:45.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea3e73d640 1 -- 172.21.15.63:0/4244593198 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea38137860 con 0x7fea380b4d60 2024-03-25T18:13:45.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.530+0000 7fea3e73d640 1 librados: init done 2024-03-25T18:13:45.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.532+0000 7fea3e73d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:45.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.532+0000 7fea3e73d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:45.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.532+0000 7fea3e73d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:13:45.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.532+0000 7fea3e73d640 1 -- 172.21.15.63:0/4244593198 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea38132c40 con 0x7fea380b4d60 2024-03-25T18:13:45.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.822+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea2c001cd0 con 0x7fea380b4d60 2024-03-25T18:13:45.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.822+0000 7fea34ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:45.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.822+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fea2c004980 con 0x7fea380b4d60 2024-03-25T18:13:45.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.822+0000 7fea34ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:45.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.822+0000 7fea34ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:45.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.822+0000 7fea34ff9640 10 monclient: dump: 2024-03-25T18:13:45.824 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:45.824 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:45.824 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:45.824 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:45.824 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:45.824 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:45.824 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:45.825 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:45.825 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:45.825 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:45.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.822+0000 7fea367fc640 4 set_mon_vals no callback set 2024-03-25T18:13:45.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.823+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fea2c004ba0 con 0x7fea380b4d60 2024-03-25T18:13:45.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.824+0000 7fea37fff640 10 monclient: get_auth_request con 0x7fea2406ded0 auth_method 0 2024-03-25T18:13:45.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.824+0000 7fea34ff9640 10 client.4276.objecter ms_handle_connect 0x7fea2406ded0 2024-03-25T18:13:45.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.828+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fea2c054740 con 0x7fea380b4d60 2024-03-25T18:13:45.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.828+0000 7fea34ff9640 10 client.4276.objecter ms_dispatch 0x7fea38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:45.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.828+0000 7fea34ff9640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:45.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.828+0000 7fea34ff9640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:45.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.828+0000 7fea34ff9640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-25T18:13:45.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.829+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea2c050080 con 0x7fea380b4d60 2024-03-25T18:13:45.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.829+0000 7fea34ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:45.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:45.829+0000 7fea34ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:46.093 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.092+0000 7fea3e73d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:46.093 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.092+0000 7fea3e73d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:46.093 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.092+0000 7fea3e73d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:13:46.093 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.092+0000 7fea3e73d640 1 -- 172.21.15.63:0/4244593198 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea380627b0 con 0x7fea380b4d60 2024-03-25T18:13:46.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.146+0000 7fea34ff9640 1 -- 172.21.15.63:0/4244593198 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fea2c052180 con 0x7fea380b4d60 2024-03-25T18:13:46.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.146+0000 7fea34ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:46.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.146+0000 7fea34ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:46.147 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:13:46.147 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:46.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.148+0000 7fea3e73d640 10 librados: watch_flush enter 2024-03-25T18:13:46.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.148+0000 7fea3e73d640 10 librados: watch_flush exit 2024-03-25T18:13:46.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.149+0000 7fea3e73d640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-25T18:13:46.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.149+0000 7fea3e73d640 10 client.4276.objecter successfully canceled tick 2024-03-25T18:13:46.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.149+0000 7fea3e73d640 1 -- 172.21.15.63:0/4244593198 >> v1:172.21.15.81:6800/3640340873 conn(0x7fea2406ded0 legacy=0x7fea24070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:46.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.149+0000 7fea3e73d640 10 monclient: shutdown 2024-03-25T18:13:46.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.149+0000 7fea3e73d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:46.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.149+0000 7fea3e73d640 1 -- 172.21.15.63:0/4244593198 >> v1:172.21.15.81:6789/0 conn(0x7fea380b4d60 legacy=0x7fea3812f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:46.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.149+0000 7fea3e73d640 1 -- 172.21.15.63:0/4244593198 shutdown_connections 2024-03-25T18:13:46.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.149+0000 7fea3e73d640 1 -- 172.21.15.63:0/4244593198 >> 172.21.15.63:0/4244593198 conn(0x7fea380617a0 msgr2=0x7fea3813c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:46.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.149+0000 7fea3e73d640 1 -- 172.21.15.63:0/4244593198 shutdown_connections 2024-03-25T18:13:46.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.149+0000 7fea3e73d640 1 -- 172.21.15.63:0/4244593198 wait complete. 2024-03-25T18:13:46.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:46.149+0000 7fea3e73d640 1 librados: shutdown 2024-03-25T18:13:46.163 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:52.969 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:53.085 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.085+0000 7f97af496640 10 monclient: get_monmap_and_config 2024-03-25T18:13:53.085 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.085+0000 7f97af496640 10 monclient: build_initial_monmap 2024-03-25T18:13:53.085 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.085+0000 7f97af496640 10 monclient: monmap: 2024-03-25T18:13:53.085 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:13:53.085 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:53.086 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:13:53.085902+0000 2024-03-25T18:13:53.086 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:13:53.085902+0000 2024-03-25T18:13:53.086 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:53.086 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:53.086 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:13:53.086 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:13:53.086 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:13:53.086 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:53.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.086+0000 7f97af496640 1 Processor -- start 2024-03-25T18:13:53.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.086+0000 7f97af496640 1 -- start start 2024-03-25T18:13:53.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.086+0000 7f97af496640 10 monclient: init 2024-03-25T18:13:53.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.087+0000 7f97af496640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:53.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.087+0000 7f97af496640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:53.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.087+0000 7f97af496640 10 monclient(hunting): picked mon.noname-b con 0x7f97a80a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:13:53.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.087+0000 7f97af496640 10 monclient(hunting): picked mon.noname-a con 0x7f97a813f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:13:53.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.087+0000 7f97af496640 10 monclient(hunting): picked mon.noname-c con 0x7f97a81430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:13:53.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.087+0000 7f97af496640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a8142520 con 0x7f97a813f710 2024-03-25T18:13:53.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.087+0000 7f97af496640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a80b28a0 con 0x7f97a81430c0 2024-03-25T18:13:53.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.087+0000 7f97af496640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a80b2a30 con 0x7f97a80a98f0 2024-03-25T18:13:53.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.087+0000 7f97af496640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:53.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.087+0000 7f97af496640 10 monclient(hunting): authenticate will time out at 7527.272686s 2024-03-25T18:13:53.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.087+0000 7f97ada33640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f97a81430c0 0x7f97a814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:49564/0 (socket says 172.21.15.63:49564) 2024-03-25T18:13:53.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.087+0000 7f97ada33640 1 -- 172.21.15.63:0/2636730419 learned_addr learned my addr 172.21.15.63:0/2636730419 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:13:53.156 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:01:34.848 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:53.165 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:01:34.857 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:53.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.176+0000 7f979ffff640 1 -- 172.21.15.63:0/2636730419 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087486694 0 0) 0x7f97a8142520 con 0x7f97a813f710 2024-03-25T18:13:53.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.176+0000 7f979ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:53.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.176+0000 7f979ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:53.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.176+0000 7f979ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:53.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.176+0000 7f979ffff640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:13:53.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.176+0000 7f979ffff640 1 -- 172.21.15.63:0/2636730419 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9794003520 con 0x7f97a813f710 2024-03-25T18:13:53.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f979ffff640 1 -- 172.21.15.63:0/2636730419 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 592865619 0 0) 0x7f9794003520 con 0x7f97a813f710 2024-03-25T18:13:53.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f979ffff640 1 -- 172.21.15.63:0/2636730419 >> v1:172.21.15.63:6790/0 conn(0x7f97a80a98f0 legacy=0x7f97a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:53.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f979ffff640 1 -- 172.21.15.63:0/2636730419 >> v1:172.21.15.81:6789/0 conn(0x7f97a81430c0 legacy=0x7f97a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:53.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f979ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:53.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f979ffff640 1 monclient: found mon.noname-a 2024-03-25T18:13:53.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f979ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:53.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f979ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:13:53.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f979ffff640 1 -- 172.21.15.63:0/2636730419 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a80b44c0 con 0x7f97a813f710 2024-03-25T18:13:53.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f979ffff640 10 monclient: _finish_auth 0 2024-03-25T18:13:53.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f979ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:53.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f979ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:53.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f979ffff640 1 -- 172.21.15.63:0/2636730419 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9798001690 con 0x7f97a813f710 2024-03-25T18:13:53.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f979ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:53.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f979ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:53.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f979ffff640 10 monclient: dump: 2024-03-25T18:13:53.194 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:53.194 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:53.194 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:53.194 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:53.194 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:53.194 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:53.194 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:53.194 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:53.194 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:53.194 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:53.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f97af496640 5 monclient: authenticate success, global_id 4269 2024-03-25T18:13:53.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.193+0000 7f97af496640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:53.220 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:01:34.913 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:53.225 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:01:34.885 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:53.235 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:01:34.927 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:53.235 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:01:34.927 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:53.236 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:01:34.896 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:53.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.250+0000 7f979ffff640 1 -- 172.21.15.63:0/2636730419 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97980017f0 con 0x7f97a813f710 2024-03-25T18:13:53.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.250+0000 7f979ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:53.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.250+0000 7f979ffff640 1 -- 172.21.15.63:0/2636730419 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9798001c00 con 0x7f97a813f710 2024-03-25T18:13:53.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.250+0000 7f979ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:53.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.251+0000 7f979ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:53.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.251+0000 7f979ffff640 10 monclient: dump: 2024-03-25T18:13:53.251 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:53.251 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:53.251 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:53.251 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:53.251 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:53.251 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:53.251 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:53.251 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:53.251 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:53.251 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:53.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.251+0000 7f97af496640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:53.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.251+0000 7f97af496640 4 set_mon_vals no callback set 2024-03-25T18:13:53.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.251+0000 7f97af496640 10 monclient: shutdown 2024-03-25T18:13:53.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.251+0000 7f97af496640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:53.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.251+0000 7f97af496640 1 -- 172.21.15.63:0/2636730419 >> v1:172.21.15.63:6789/0 conn(0x7f97a813f710 legacy=0x7f97a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:53.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.251+0000 7f97af496640 1 -- 172.21.15.63:0/2636730419 shutdown_connections 2024-03-25T18:13:53.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.251+0000 7f97af496640 1 -- 172.21.15.63:0/2636730419 >> 172.21.15.63:0/2636730419 conn(0x7f97a80b7ea0 msgr2=0x7f97a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:53.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.251+0000 7f97af496640 1 -- 172.21.15.63:0/2636730419 shutdown_connections 2024-03-25T18:13:53.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.251+0000 7f97af496640 1 -- 172.21.15.63:0/2636730419 wait complete. 2024-03-25T18:13:53.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.252+0000 7f97af496640 10 monclient: build_initial_monmap 2024-03-25T18:13:53.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.252+0000 7f97af496640 10 monclient: monmap: 2024-03-25T18:13:53.254 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:13:53.254 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:53.254 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:13:53.254 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:13:53.255 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:53.255 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:13:53.255 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:13:53.255 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:13:53.255 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:13:53.255 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:53.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.252+0000 7f97af496640 1 librados: starting msgr at 2024-03-25T18:13:53.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.252+0000 7f97af496640 1 librados: starting objecter 2024-03-25T18:13:53.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.253+0000 7f97af496640 1 Processor -- start 2024-03-25T18:13:53.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.253+0000 7f97af496640 1 -- start start 2024-03-25T18:13:53.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.253+0000 7f97af496640 1 librados: setting wanted keys 2024-03-25T18:13:53.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.253+0000 7f97af496640 1 librados: calling monclient init 2024-03-25T18:13:53.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.253+0000 7f97af496640 10 monclient: init 2024-03-25T18:13:53.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.253+0000 7f97af496640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:53.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.253+0000 7f97af496640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:53.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.253+0000 7f97af496640 10 monclient(hunting): picked mon.noname-b con 0x7f97a80a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:13:53.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.253+0000 7f97af496640 10 monclient(hunting): picked mon.noname-a con 0x7f97a813f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:13:53.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.253+0000 7f97af496640 10 monclient(hunting): picked mon.noname-c con 0x7f97a81430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:13:53.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.253+0000 7f97af496640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a80abd70 con 0x7f97a813f710 2024-03-25T18:13:53.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.253+0000 7f97af496640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a81d7710 con 0x7f97a80a98f0 2024-03-25T18:13:53.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.253+0000 7f97af496640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a81d8340 con 0x7f97a81430c0 2024-03-25T18:13:53.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.253+0000 7f97af496640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:53.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.253+0000 7f97af496640 10 monclient(hunting): authenticate will time out at 7527.439036s 2024-03-25T18:13:53.287 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.288+0000 7f97ad232640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f97a80a98f0 0x7f97a8137be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:49574/0 (socket says 172.21.15.63:49574) 2024-03-25T18:13:53.287 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.288+0000 7f97ad232640 1 -- 172.21.15.63:0/1027101527 learned_addr learned my addr 172.21.15.63:0/1027101527 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:13:53.291 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:01:34.952 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:53.292 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:01:34.985 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:53.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.296+0000 7f979dffb640 10 client.?.objecter ms_handle_connect 0x7f97a81430c0 2024-03-25T18:13:53.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.297+0000 7f979dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:53.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.303+0000 7f979dffb640 10 client.?.objecter ms_handle_connect 0x7f97a813f710 2024-03-25T18:13:53.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.303+0000 7f979dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:53.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.312+0000 7f979dffb640 10 client.?.objecter ms_handle_connect 0x7f97a80a98f0 2024-03-25T18:13:53.335 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.312+0000 7f979dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:53.335 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.314+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619674159 0 0) 0x7f97a81d8340 con 0x7f97a81430c0 2024-03-25T18:13:53.335 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.314+0000 7f979dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:53.335 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.314+0000 7f979dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:53.335 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.314+0000 7f979dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:53.335 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.314+0000 7f979dffb640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:13:53.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.314+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f978c004eb0 con 0x7f97a81430c0 2024-03-25T18:13:53.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.323+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472916806 0 0) 0x7f97a80abd70 con 0x7f97a813f710 2024-03-25T18:13:53.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.323+0000 7f979dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:53.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.323+0000 7f979dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:53.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.323+0000 7f979dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:53.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.323+0000 7f979dffb640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:13:53.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.323+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97a81d8340 con 0x7f97a813f710 2024-03-25T18:13:53.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.342+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312848030 0 0) 0x7f97a81d7710 con 0x7f97a80a98f0 2024-03-25T18:13:53.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.343+0000 7f979dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:53.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.343+0000 7f979dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:53.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.343+0000 7f979dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:53.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.343+0000 7f979dffb640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:13:53.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.343+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97a80abd70 con 0x7f97a80a98f0 2024-03-25T18:13:53.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.345+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3914004889 0 0) 0x7f97a81d8340 con 0x7f97a813f710 2024-03-25T18:13:53.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.345+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97a81d7710 con 0x7f97a813f710 2024-03-25T18:13:53.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.345+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9798001b80 con 0x7f97a813f710 2024-03-25T18:13:53.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.345+0000 7f979dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:53.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.345+0000 7f979dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:53.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.345+0000 7f979dffb640 10 monclient(hunting): dump: 2024-03-25T18:13:53.345 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:53.345 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:53.345 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:53.346 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:53.346 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:53.346 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:53.346 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:53.346 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:53.346 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:53.346 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:53.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.345+0000 7f979dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:53.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.346+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1390280021 0 0) 0x7f978c004eb0 con 0x7f97a81430c0 2024-03-25T18:13:53.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.346+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97a81d8340 con 0x7f97a81430c0 2024-03-25T18:13:53.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.346+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f97a0001b00 con 0x7f97a81430c0 2024-03-25T18:13:53.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.346+0000 7f979dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:53.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.346+0000 7f979dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:53.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.346+0000 7f979dffb640 10 monclient(hunting): dump: 2024-03-25T18:13:53.347 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:53.347 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:53.347 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:53.348 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:53.348 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:53.348 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:53.348 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:53.348 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:53.348 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:53.348 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:53.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3203061052 0 0) 0x7f97a81d7710 con 0x7f97a813f710 2024-03-25T18:13:53.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 >> v1:172.21.15.63:6790/0 conn(0x7f97a81430c0 legacy=0x7f97a81d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:53.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 >> v1:172.21.15.81:6789/0 conn(0x7f97a80a98f0 legacy=0x7f97a8137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:53.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f979dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:53.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f979dffb640 1 monclient: found mon.a 2024-03-25T18:13:53.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f979dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:53.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f979dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:53.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a81d84d0 con 0x7f97a813f710 2024-03-25T18:13:53.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f979dffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:53.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f979dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:53.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f979dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:53.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f97af496640 5 monclient: authenticate success, global_id 4272 2024-03-25T18:13:53.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f97af496640 10 monclient: _renew_subs 2024-03-25T18:13:53.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f97af496640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:53.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f97af496640 1 -- 172.21.15.63:0/1027101527 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a81d41e0 con 0x7f97a813f710 2024-03-25T18:13:53.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f97af496640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:53.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.371+0000 7f97af496640 10 monclient: _renew_subs 2024-03-25T18:13:53.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.372+0000 7f97af496640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:53.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.372+0000 7f97af496640 1 -- 172.21.15.63:0/1027101527 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a81d5150 con 0x7f97a813f710 2024-03-25T18:13:53.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.372+0000 7f97af496640 1 librados: init done 2024-03-25T18:13:53.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.372+0000 7f97af496640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:53.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.372+0000 7f97af496640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:53.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.372+0000 7f97af496640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:53.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.372+0000 7f97af496640 1 -- 172.21.15.63:0/1027101527 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a81d7950 con 0x7f97a813f710 2024-03-25T18:13:53.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.703+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97980038a0 con 0x7f97a813f710 2024-03-25T18:13:53.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.703+0000 7f979dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:53.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.703+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9798003a10 con 0x7f97a813f710 2024-03-25T18:13:53.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.703+0000 7f979dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:53.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.703+0000 7f979dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:53.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.703+0000 7f979dffb640 10 monclient: dump: 2024-03-25T18:13:53.703 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:13:53.703 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:13:53.703 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:53.703 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:13:53.703 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:53.703 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:13:53.703 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:13:53.704 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:13:53.704 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:13:53.704 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:13:53.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.703+0000 7f979f7fe640 4 set_mon_vals no callback set 2024-03-25T18:13:53.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.703+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f97980199e0 con 0x7f97a813f710 2024-03-25T18:13:53.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.704+0000 7f97ad232640 10 monclient: get_auth_request con 0x7f978c06df70 auth_method 0 2024-03-25T18:13:53.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.704+0000 7f979dffb640 10 client.4272.objecter ms_handle_connect 0x7f978c06df70 2024-03-25T18:13:53.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.714+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f9798053560 con 0x7f97a813f710 2024-03-25T18:13:53.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.714+0000 7f979dffb640 10 client.4272.objecter ms_dispatch 0x7f97a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:53.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.714+0000 7f979dffb640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:53.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.714+0000 7f979dffb640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:53.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.714+0000 7f979dffb640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-25T18:13:53.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.714+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f979804f070 con 0x7f97a813f710 2024-03-25T18:13:53.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.714+0000 7f979dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:53.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.714+0000 7f979dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:53.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.974+0000 7f97af496640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:53.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.974+0000 7f97af496640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:53.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.974+0000 7f97af496640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:13:53.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:53.974+0000 7f97af496640 1 -- 172.21.15.63:0/1027101527 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97a8074cc0 con 0x7f97a813f710 2024-03-25T18:13:54.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:54.016+0000 7f979dffb640 1 -- 172.21.15.63:0/1027101527 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f9798050fa0 con 0x7f97a813f710 2024-03-25T18:13:54.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:54.016+0000 7f979dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:54.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:54.016+0000 7f979dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:54.016 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:13:54.016 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:54.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:54.018+0000 7f97af496640 10 librados: watch_flush enter 2024-03-25T18:13:54.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:54.018+0000 7f97af496640 10 librados: watch_flush exit 2024-03-25T18:13:54.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:54.018+0000 7f97af496640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-25T18:13:54.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:54.018+0000 7f97af496640 10 client.4272.objecter successfully canceled tick 2024-03-25T18:13:54.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:54.018+0000 7f97af496640 1 -- 172.21.15.63:0/1027101527 >> v1:172.21.15.81:6800/3640340873 conn(0x7f978c06df70 legacy=0x7f978c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:54.018+0000 7f97af496640 10 monclient: shutdown 2024-03-25T18:13:54.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:54.018+0000 7f97af496640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:54.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:54.018+0000 7f97af496640 1 -- 172.21.15.63:0/1027101527 >> v1:172.21.15.63:6789/0 conn(0x7f97a813f710 legacy=0x7f97a81d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:54.019+0000 7f97af496640 1 -- 172.21.15.63:0/1027101527 shutdown_connections 2024-03-25T18:13:54.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:54.019+0000 7f97af496640 1 -- 172.21.15.63:0/1027101527 >> 172.21.15.63:0/1027101527 conn(0x7f97a80b7ea0 msgr2=0x7f97a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:54.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:54.019+0000 7f97af496640 1 -- 172.21.15.63:0/1027101527 shutdown_connections 2024-03-25T18:13:54.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:54.019+0000 7f97af496640 1 -- 172.21.15.63:0/1027101527 wait complete. 2024-03-25T18:13:54.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:13:54.019+0000 7f97af496640 1 librados: shutdown 2024-03-25T18:13:54.031 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:00.837 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:00.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.951+0000 7f31c1cc1640 10 monclient: get_monmap_and_config 2024-03-25T18:14:00.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.951+0000 7f31c1cc1640 10 monclient: build_initial_monmap 2024-03-25T18:14:00.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.951+0000 7f31c1cc1640 10 monclient: monmap: 2024-03-25T18:14:00.952 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:14:00.952 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:00.952 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:14:00.952427+0000 2024-03-25T18:14:00.952 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:14:00.952427+0000 2024-03-25T18:14:00.952 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:00.952 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:00.952 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:14:00.952 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:14:00.953 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:14:00.953 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:00.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.952+0000 7f31c1cc1640 1 Processor -- start 2024-03-25T18:14:00.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.952+0000 7f31c1cc1640 1 -- start start 2024-03-25T18:14:00.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.952+0000 7f31c1cc1640 10 monclient: init 2024-03-25T18:14:00.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.952+0000 7f31c1cc1640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:00.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.952+0000 7f31c1cc1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:00.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.952+0000 7f31c1cc1640 10 monclient(hunting): picked mon.noname-b con 0x7f31bc0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:14:00.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.952+0000 7f31c1cc1640 10 monclient(hunting): picked mon.noname-c con 0x7f31bc13f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:14:00.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.952+0000 7f31c1cc1640 10 monclient(hunting): picked mon.noname-a con 0x7f31bc1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:14:00.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.952+0000 7f31c1cc1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc142520 con 0x7f31bc1430c0 2024-03-25T18:14:00.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.952+0000 7f31c1cc1640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc0b28a0 con 0x7f31bc13f710 2024-03-25T18:14:00.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.952+0000 7f31c1cc1640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc0b2a30 con 0x7f31bc0a98f0 2024-03-25T18:14:00.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.952+0000 7f31c1cc1640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:00.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.952+0000 7f31c1cc1640 10 monclient(hunting): authenticate will time out at 7535.139147s 2024-03-25T18:14:00.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.960+0000 7f31bbfff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f31bc1430c0 0x7f31bc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:37966/0 (socket says 172.21.15.63:37966) 2024-03-25T18:14:00.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.960+0000 7f31bbfff640 1 -- 172.21.15.63:0/3803947012 learned_addr learned my addr 172.21.15.63:0/3803947012 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:14:00.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.974+0000 7f31ba7fc640 1 -- 172.21.15.63:0/3803947012 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185690263 0 0) 0x7f31bc142520 con 0x7f31bc1430c0 2024-03-25T18:14:00.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.974+0000 7f31ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:00.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.974+0000 7f31ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:00.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.974+0000 7f31ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:00.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.974+0000 7f31ba7fc640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:14:00.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.974+0000 7f31ba7fc640 1 -- 172.21.15.63:0/3803947012 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31a4003520 con 0x7f31bc1430c0 2024-03-25T18:14:00.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.990+0000 7f31ba7fc640 1 -- 172.21.15.63:0/3803947012 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686514724 0 0) 0x7f31bc0b28a0 con 0x7f31bc13f710 2024-03-25T18:14:00.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.990+0000 7f31ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:00.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.990+0000 7f31ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:00.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.990+0000 7f31ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:00.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.990+0000 7f31ba7fc640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:14:00.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.990+0000 7f31ba7fc640 1 -- 172.21.15.63:0/3803947012 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31bc142520 con 0x7f31bc13f710 2024-03-25T18:14:01.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31ba7fc640 1 -- 172.21.15.63:0/3803947012 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2417714044 0 0) 0x7f31a4003520 con 0x7f31bc1430c0 2024-03-25T18:14:01.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31ba7fc640 1 -- 172.21.15.63:0/3803947012 >> v1:172.21.15.63:6790/0 conn(0x7f31bc0a98f0 legacy=0x7f31bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:01.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31ba7fc640 1 -- 172.21.15.63:0/3803947012 >> v1:172.21.15.81:6789/0 conn(0x7f31bc13f710 legacy=0x7f31bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:01.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31ba7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:01.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31ba7fc640 1 monclient: found mon.noname-a 2024-03-25T18:14:01.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:01.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31ba7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:14:01.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31ba7fc640 1 -- 172.21.15.63:0/3803947012 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc0b44c0 con 0x7f31bc1430c0 2024-03-25T18:14:01.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31ba7fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:01.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31ba7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:01.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:01.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31ba7fc640 1 -- 172.21.15.63:0/3803947012 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f31b4001a10 con 0x7f31bc1430c0 2024-03-25T18:14:01.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31c1cc1640 5 monclient: authenticate success, global_id 4275 2024-03-25T18:14:01.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31c1cc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:01.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:01.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31ba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:01.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31ba7fc640 10 monclient: dump: 2024-03-25T18:14:01.002 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:01.002 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:01.002 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:01.002 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:01.002 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:01.002 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:01.002 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:01.002 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:01.002 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:01.002 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:01.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:00.999+0000 7f31c1cc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:01.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.027+0000 7f31ba7fc640 1 -- 172.21.15.63:0/3803947012 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31b4001b70 con 0x7f31bc1430c0 2024-03-25T18:14:01.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.027+0000 7f31ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:01.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.027+0000 7f31ba7fc640 1 -- 172.21.15.63:0/3803947012 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f31b4001f80 con 0x7f31bc1430c0 2024-03-25T18:14:01.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.027+0000 7f31c1cc1640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:01.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.027+0000 7f31c1cc1640 4 set_mon_vals no callback set 2024-03-25T18:14:01.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.027+0000 7f31ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:01.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.027+0000 7f31ba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:01.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.027+0000 7f31ba7fc640 10 monclient: dump: 2024-03-25T18:14:01.028 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:01.028 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:01.028 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:01.028 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:01.028 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:01.028 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:01.029 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:01.029 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:01.029 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:01.029 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:01.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.027+0000 7f31c1cc1640 10 monclient: shutdown 2024-03-25T18:14:01.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.027+0000 7f31c1cc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:01.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.027+0000 7f31c1cc1640 1 -- 172.21.15.63:0/3803947012 >> v1:172.21.15.63:6789/0 conn(0x7f31bc1430c0 legacy=0x7f31bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:01.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.028+0000 7f31c1cc1640 1 -- 172.21.15.63:0/3803947012 shutdown_connections 2024-03-25T18:14:01.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.028+0000 7f31c1cc1640 1 -- 172.21.15.63:0/3803947012 >> 172.21.15.63:0/3803947012 conn(0x7f31bc0b7ea0 msgr2=0x7f31bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:01.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.028+0000 7f31c1cc1640 1 -- 172.21.15.63:0/3803947012 shutdown_connections 2024-03-25T18:14:01.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.028+0000 7f31c1cc1640 1 -- 172.21.15.63:0/3803947012 wait complete. 2024-03-25T18:14:01.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.028+0000 7f31c1cc1640 10 monclient: build_initial_monmap 2024-03-25T18:14:01.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.028+0000 7f31c1cc1640 10 monclient: monmap: 2024-03-25T18:14:01.031 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:14:01.031 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:01.031 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:14:01.031 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:14:01.031 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:01.031 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:14:01.031 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:14:01.031 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:14:01.031 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:14:01.031 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:01.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.029+0000 7f31c1cc1640 1 librados: starting msgr at 2024-03-25T18:14:01.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.029+0000 7f31c1cc1640 1 librados: starting objecter 2024-03-25T18:14:01.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.029+0000 7f31c1cc1640 1 Processor -- start 2024-03-25T18:14:01.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.029+0000 7f31c1cc1640 1 -- start start 2024-03-25T18:14:01.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.029+0000 7f31c1cc1640 1 librados: setting wanted keys 2024-03-25T18:14:01.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.029+0000 7f31c1cc1640 1 librados: calling monclient init 2024-03-25T18:14:01.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.029+0000 7f31c1cc1640 10 monclient: init 2024-03-25T18:14:01.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.030+0000 7f31c1cc1640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:01.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.030+0000 7f31c1cc1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:01.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.030+0000 7f31c1cc1640 10 monclient(hunting): picked mon.noname-b con 0x7f31bc0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:14:01.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.030+0000 7f31c1cc1640 10 monclient(hunting): picked mon.noname-a con 0x7f31bc13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:14:01.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.030+0000 7f31c1cc1640 10 monclient(hunting): picked mon.noname-c con 0x7f31bc1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:14:01.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.030+0000 7f31c1cc1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc136c50 con 0x7f31bc13f710 2024-03-25T18:14:01.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.030+0000 7f31c1cc1640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc136ec0 con 0x7f31bc0a98f0 2024-03-25T18:14:01.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.030+0000 7f31c1cc1640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc1317f0 con 0x7f31bc1430c0 2024-03-25T18:14:01.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.030+0000 7f31c1cc1640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:01.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.030+0000 7f31c1cc1640 10 monclient(hunting): authenticate will time out at 7535.216712s 2024-03-25T18:14:01.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.033+0000 7f31baffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f31bc13f710 0x7f31bc1dac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:38026/0 (socket says 172.21.15.63:38026) 2024-03-25T18:14:01.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.033+0000 7f31baffd640 1 -- 172.21.15.63:0/1272754799 learned_addr learned my addr 172.21.15.63:0/1272754799 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:14:01.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.034+0000 7f31a3fff640 10 client.?.objecter ms_handle_connect 0x7f31bc13f710 2024-03-25T18:14:01.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.034+0000 7f31a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:01.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.048+0000 7f31a3fff640 10 client.?.objecter ms_handle_connect 0x7f31bc1430c0 2024-03-25T18:14:01.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.048+0000 7f31a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:01.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.052+0000 7f31a3fff640 10 client.?.objecter ms_handle_connect 0x7f31bc0a98f0 2024-03-25T18:14:01.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.052+0000 7f31a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:01.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.053+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527648575 0 0) 0x7f31bc1317f0 con 0x7f31bc1430c0 2024-03-25T18:14:01.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.053+0000 7f31a3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:01.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.053+0000 7f31a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:01.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.053+0000 7f31a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:01.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.053+0000 7f31a3fff640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:14:01.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.053+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31a8004eb0 con 0x7f31bc1430c0 2024-03-25T18:14:01.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.057+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009137061 0 0) 0x7f31bc136c50 con 0x7f31bc13f710 2024-03-25T18:14:01.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.057+0000 7f31a3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:01.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.057+0000 7f31a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:01.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.057+0000 7f31a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:01.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.057+0000 7f31a3fff640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:14:01.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.057+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31bc1317f0 con 0x7f31bc13f710 2024-03-25T18:14:01.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.060+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681941230 0 0) 0x7f31bc136ec0 con 0x7f31bc0a98f0 2024-03-25T18:14:01.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.060+0000 7f31a3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:01.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.060+0000 7f31a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:01.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.060+0000 7f31a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:01.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.060+0000 7f31a3fff640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:14:01.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.060+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31bc136c50 con 0x7f31bc0a98f0 2024-03-25T18:14:01.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.065+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2218420418 0 0) 0x7f31a8004eb0 con 0x7f31bc1430c0 2024-03-25T18:14:01.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.065+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31bc136ec0 con 0x7f31bc1430c0 2024-03-25T18:14:01.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.065+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f31b4001e80 con 0x7f31bc1430c0 2024-03-25T18:14:01.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.065+0000 7f31a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:01.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.065+0000 7f31a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:01.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.065+0000 7f31a3fff640 10 monclient(hunting): dump: 2024-03-25T18:14:01.066 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:01.066 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:01.066 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:01.066 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:01.066 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:01.066 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:01.066 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:01.067 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:01.067 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:01.067 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:01.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.065+0000 7f31a3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:01.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.077+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4258834063 0 0) 0x7f31bc136ec0 con 0x7f31bc1430c0 2024-03-25T18:14:01.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.077+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 >> v1:172.21.15.81:6789/0 conn(0x7f31bc0a98f0 legacy=0x7f31bc1310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:01.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.077+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 >> v1:172.21.15.63:6789/0 conn(0x7f31bc13f710 legacy=0x7f31bc1dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:01.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.077+0000 7f31a3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:01.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.077+0000 7f31a3fff640 1 monclient: found mon.c 2024-03-25T18:14:01.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.078+0000 7f31a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:01.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.078+0000 7f31a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:14:01.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.078+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc131980 con 0x7f31bc1430c0 2024-03-25T18:14:01.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.078+0000 7f31a3fff640 10 monclient: _finish_auth 0 2024-03-25T18:14:01.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.078+0000 7f31a3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:01.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.078+0000 7f31a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:01.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.078+0000 7f31c1cc1640 5 monclient: authenticate success, global_id 4289 2024-03-25T18:14:01.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.078+0000 7f31c1cc1640 10 monclient: _renew_subs 2024-03-25T18:14:01.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.078+0000 7f31c1cc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:14:01.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.078+0000 7f31c1cc1640 1 -- 172.21.15.63:0/1272754799 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31bc135070 con 0x7f31bc1430c0 2024-03-25T18:14:01.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.078+0000 7f31c1cc1640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:01.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.078+0000 7f31c1cc1640 10 monclient: _renew_subs 2024-03-25T18:14:01.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.078+0000 7f31c1cc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:14:01.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.078+0000 7f31c1cc1640 1 -- 172.21.15.63:0/1272754799 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31bc0b7870 con 0x7f31bc1430c0 2024-03-25T18:14:01.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.078+0000 7f31c1cc1640 1 librados: init done 2024-03-25T18:14:01.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.079+0000 7f31c1cc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:01.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.079+0000 7f31c1cc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:01.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.079+0000 7f31c1cc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:14:01.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.079+0000 7f31c1cc1640 1 -- 172.21.15.63:0/1272754799 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31bc0627b0 con 0x7f31bc1430c0 2024-03-25T18:14:01.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.387+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31b4002040 con 0x7f31bc1430c0 2024-03-25T18:14:01.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.387+0000 7f31a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:01.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.387+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f31b4004fb0 con 0x7f31bc1430c0 2024-03-25T18:14:01.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.387+0000 7f31a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:01.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.387+0000 7f31a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:01.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.387+0000 7f31a3fff640 10 monclient: dump: 2024-03-25T18:14:01.388 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:01.389 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:01.389 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:01.389 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:01.389 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:01.389 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:01.389 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:01.389 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:01.389 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:01.389 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:01.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.387+0000 7f31b9ffb640 4 set_mon_vals no callback set 2024-03-25T18:14:01.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.388+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f31b401af80 con 0x7f31bc1430c0 2024-03-25T18:14:01.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.388+0000 7f31bb7fe640 10 monclient: get_auth_request con 0x7f31a806ddf0 auth_method 0 2024-03-25T18:14:01.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.389+0000 7f31a3fff640 10 client.4289.objecter ms_handle_connect 0x7f31a806ddf0 2024-03-25T18:14:01.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.395+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f31b4054af0 con 0x7f31bc1430c0 2024-03-25T18:14:01.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.395+0000 7f31a3fff640 10 client.4289.objecter ms_dispatch 0x7f31bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:01.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.395+0000 7f31a3fff640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:01.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.395+0000 7f31a3fff640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:01.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.395+0000 7f31a3fff640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-25T18:14:01.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.395+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f31b4050600 con 0x7f31bc1430c0 2024-03-25T18:14:01.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.395+0000 7f31a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:01.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.395+0000 7f31a3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:01.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.654+0000 7f31c1cc1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:01.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.654+0000 7f31c1cc1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:01.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.654+0000 7f31c1cc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:14:01.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.654+0000 7f31c1cc1640 1 -- 172.21.15.63:0/1272754799 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31bc0b7ac0 con 0x7f31bc1430c0 2024-03-25T18:14:01.818 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.818+0000 7f31a3fff640 1 -- 172.21.15.63:0/1272754799 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f31b4052530 con 0x7f31bc1430c0 2024-03-25T18:14:01.819 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.818+0000 7f31a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:01.819 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.818+0000 7f31a3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:01.819 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:14:01.819 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:01.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.820+0000 7f31c1cc1640 10 librados: watch_flush enter 2024-03-25T18:14:01.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.820+0000 7f31c1cc1640 10 librados: watch_flush exit 2024-03-25T18:14:01.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.820+0000 7f31c1cc1640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-25T18:14:01.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.820+0000 7f31c1cc1640 10 client.4289.objecter successfully canceled tick 2024-03-25T18:14:01.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.821+0000 7f31c1cc1640 1 -- 172.21.15.63:0/1272754799 >> v1:172.21.15.81:6800/3640340873 conn(0x7f31a806ddf0 legacy=0x7f31a80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:01.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.821+0000 7f31c1cc1640 10 monclient: shutdown 2024-03-25T18:14:01.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.821+0000 7f31c1cc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:01.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.821+0000 7f31c1cc1640 1 -- 172.21.15.63:0/1272754799 >> v1:172.21.15.63:6790/0 conn(0x7f31bc1430c0 legacy=0x7f31bc1db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:01.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.821+0000 7f31c1cc1640 1 -- 172.21.15.63:0/1272754799 shutdown_connections 2024-03-25T18:14:01.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.821+0000 7f31c1cc1640 1 -- 172.21.15.63:0/1272754799 >> 172.21.15.63:0/1272754799 conn(0x7f31bc0b7ea0 msgr2=0x7f31bc13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:01.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.821+0000 7f31c1cc1640 1 -- 172.21.15.63:0/1272754799 shutdown_connections 2024-03-25T18:14:01.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.821+0000 7f31c1cc1640 1 -- 172.21.15.63:0/1272754799 wait complete. 2024-03-25T18:14:01.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:01.821+0000 7f31c1cc1640 1 librados: shutdown 2024-03-25T18:14:01.834 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:08.640 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:08.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.753+0000 7f67acd23640 10 monclient: get_monmap_and_config 2024-03-25T18:14:08.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.753+0000 7f67acd23640 10 monclient: build_initial_monmap 2024-03-25T18:14:08.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.753+0000 7f67acd23640 10 monclient: monmap: 2024-03-25T18:14:08.755 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:14:08.755 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:08.755 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:14:08.754840+0000 2024-03-25T18:14:08.755 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:14:08.754840+0000 2024-03-25T18:14:08.755 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:08.755 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:08.755 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:14:08.755 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:14:08.755 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:14:08.755 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:08.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.754+0000 7f67acd23640 1 Processor -- start 2024-03-25T18:14:08.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.754+0000 7f67acd23640 1 -- start start 2024-03-25T18:14:08.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.754+0000 7f67acd23640 10 monclient: init 2024-03-25T18:14:08.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.755+0000 7f67acd23640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:08.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.755+0000 7f67acd23640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:08.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.755+0000 7f67acd23640 10 monclient(hunting): picked mon.noname-a con 0x7f67a80a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:14:08.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.755+0000 7f67acd23640 10 monclient(hunting): picked mon.noname-c con 0x7f67a813f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:14:08.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.755+0000 7f67acd23640 10 monclient(hunting): picked mon.noname-b con 0x7f67a81430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:14:08.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.755+0000 7f67acd23640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a8142520 con 0x7f67a80a98f0 2024-03-25T18:14:08.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.755+0000 7f67acd23640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a80b28a0 con 0x7f67a813f710 2024-03-25T18:14:08.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.755+0000 7f67acd23640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a80b2a30 con 0x7f67a81430c0 2024-03-25T18:14:08.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.755+0000 7f67acd23640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:08.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.755+0000 7f67acd23640 10 monclient(hunting): authenticate will time out at 7542.942163s 2024-03-25T18:14:08.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.757+0000 7f679ffff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f67a813f710 0x7f67a81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:60572/0 (socket says 172.21.15.63:60572) 2024-03-25T18:14:08.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.757+0000 7f679ffff640 1 -- 172.21.15.63:0/313972041 learned_addr learned my addr 172.21.15.63:0/313972041 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:14:08.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.775+0000 7f67a5d9b640 1 -- 172.21.15.63:0/313972041 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046736221 0 0) 0x7f67a80b28a0 con 0x7f67a813f710 2024-03-25T18:14:08.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.775+0000 7f67a5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:08.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.775+0000 7f67a5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:08.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.775+0000 7f67a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:08.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.775+0000 7f67a5d9b640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:14:08.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.775+0000 7f67a5d9b640 1 -- 172.21.15.63:0/313972041 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6794003520 con 0x7f67a813f710 2024-03-25T18:14:08.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67a5d9b640 1 -- 172.21.15.63:0/313972041 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 177675297 0 0) 0x7f6794003520 con 0x7f67a813f710 2024-03-25T18:14:08.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67a5d9b640 1 -- 172.21.15.63:0/313972041 >> v1:172.21.15.63:6790/0 conn(0x7f67a81430c0 legacy=0x7f67a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67a5d9b640 1 -- 172.21.15.63:0/313972041 >> v1:172.21.15.63:6789/0 conn(0x7f67a80a98f0 legacy=0x7f67a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67a5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:14:08.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67a5d9b640 1 monclient: found mon.noname-c 2024-03-25T18:14:08.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67a5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:08.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67a5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:14:08.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67a5d9b640 1 -- 172.21.15.63:0/313972041 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a80b44c0 con 0x7f67a813f710 2024-03-25T18:14:08.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67a5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:14:08.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67a5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:14:08.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67a5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:08.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67a5d9b640 1 -- 172.21.15.63:0/313972041 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6790001690 con 0x7f67a813f710 2024-03-25T18:14:08.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67a5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:08.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67a5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:08.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67a5d9b640 10 monclient: dump: 2024-03-25T18:14:08.796 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:08.796 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:08.797 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:08.797 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:08.797 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:08.797 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:08.797 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:08.797 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:08.797 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:08.797 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:08.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67acd23640 5 monclient: authenticate success, global_id 4291 2024-03-25T18:14:08.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.794+0000 7f67acd23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:08.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.822+0000 7f67a5d9b640 1 -- 172.21.15.63:0/313972041 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67900017f0 con 0x7f67a813f710 2024-03-25T18:14:08.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.822+0000 7f67a5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:08.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.822+0000 7f67a5d9b640 1 -- 172.21.15.63:0/313972041 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6790001c00 con 0x7f67a813f710 2024-03-25T18:14:08.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.822+0000 7f67a5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:08.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.822+0000 7f67a5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:08.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.822+0000 7f67a5d9b640 10 monclient: dump: 2024-03-25T18:14:08.823 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:08.823 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:08.823 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:08.823 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:08.823 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:08.823 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:08.824 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:08.824 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:08.824 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:08.824 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:08.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.822+0000 7f67acd23640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:08.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.822+0000 7f67acd23640 4 set_mon_vals no callback set 2024-03-25T18:14:08.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.822+0000 7f67acd23640 10 monclient: shutdown 2024-03-25T18:14:08.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.822+0000 7f67acd23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:08.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.822+0000 7f67acd23640 1 -- 172.21.15.63:0/313972041 >> v1:172.21.15.81:6789/0 conn(0x7f67a813f710 legacy=0x7f67a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.822+0000 7f67acd23640 1 -- 172.21.15.63:0/313972041 shutdown_connections 2024-03-25T18:14:08.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.822+0000 7f67acd23640 1 -- 172.21.15.63:0/313972041 >> 172.21.15.63:0/313972041 conn(0x7f67a80b7ea0 msgr2=0x7f67a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:08.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.822+0000 7f67acd23640 1 -- 172.21.15.63:0/313972041 shutdown_connections 2024-03-25T18:14:08.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.822+0000 7f67acd23640 1 -- 172.21.15.63:0/313972041 wait complete. 2024-03-25T18:14:08.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.823+0000 7f67acd23640 10 monclient: build_initial_monmap 2024-03-25T18:14:08.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.823+0000 7f67acd23640 10 monclient: monmap: 2024-03-25T18:14:08.827 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:14:08.827 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:08.827 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:14:08.827 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:14:08.827 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:08.827 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:14:08.827 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:14:08.827 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:14:08.827 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:14:08.827 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:08.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 1 librados: starting msgr at 2024-03-25T18:14:08.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 1 librados: starting objecter 2024-03-25T18:14:08.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 1 Processor -- start 2024-03-25T18:14:08.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 1 -- start start 2024-03-25T18:14:08.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 1 librados: setting wanted keys 2024-03-25T18:14:08.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 1 librados: calling monclient init 2024-03-25T18:14:08.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 10 monclient: init 2024-03-25T18:14:08.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:08.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:08.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 10 monclient(hunting): picked mon.noname-a con 0x7f67a80a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:14:08.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 10 monclient(hunting): picked mon.noname-c con 0x7f67a813f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:14:08.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 10 monclient(hunting): picked mon.noname-b con 0x7f67a81430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:14:08.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a80b1870 con 0x7f67a80a98f0 2024-03-25T18:14:08.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a80ac910 con 0x7f67a81430c0 2024-03-25T18:14:08.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a80abb00 con 0x7f67a813f710 2024-03-25T18:14:08.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:08.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.824+0000 7f67acd23640 10 monclient(hunting): authenticate will time out at 7543.011165s 2024-03-25T18:14:08.833 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.832+0000 7f67a6d9d640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f67a81430c0 0x7f67a81e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:60630/0 (socket says 172.21.15.63:60630) 2024-03-25T18:14:08.833 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.832+0000 7f67a6d9d640 1 -- 172.21.15.63:0/2004077306 learned_addr learned my addr 172.21.15.63:0/2004077306 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:14:08.834 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.833+0000 7f679effd640 10 client.?.objecter ms_handle_connect 0x7f67a81430c0 2024-03-25T18:14:08.834 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.833+0000 7f679effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:08.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.835+0000 7f679effd640 10 client.?.objecter ms_handle_connect 0x7f67a80a98f0 2024-03-25T18:14:08.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.835+0000 7f679effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:08.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.839+0000 7f679effd640 10 client.?.objecter ms_handle_connect 0x7f67a813f710 2024-03-25T18:14:08.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.839+0000 7f679effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:08.845 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.845+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510114978 0 0) 0x7f67a80ac910 con 0x7f67a81430c0 2024-03-25T18:14:08.845 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.845+0000 7f679effd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:08.846 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.845+0000 7f679effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:08.846 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.845+0000 7f679effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:08.846 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.845+0000 7f679effd640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:14:08.846 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.845+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6788004eb0 con 0x7f67a81430c0 2024-03-25T18:14:08.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.848+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135067904 0 0) 0x7f67a8142520 con 0x7f67a80a98f0 2024-03-25T18:14:08.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.848+0000 7f679effd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:08.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.848+0000 7f679effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:08.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.848+0000 7f679effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:08.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.848+0000 7f679effd640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:14:08.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.848+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67a80ac910 con 0x7f67a80a98f0 2024-03-25T18:14:08.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.849+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293782306 0 0) 0x7f67a80abb00 con 0x7f67a813f710 2024-03-25T18:14:08.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.849+0000 7f679effd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:08.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.849+0000 7f679effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:08.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.849+0000 7f679effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:08.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.849+0000 7f679effd640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:14:08.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.849+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67a8142520 con 0x7f67a813f710 2024-03-25T18:14:08.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.857+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2653178669 0 0) 0x7f67a80ac910 con 0x7f67a80a98f0 2024-03-25T18:14:08.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.857+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67a80abb00 con 0x7f67a80a98f0 2024-03-25T18:14:08.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.857+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6798001700 con 0x7f67a80a98f0 2024-03-25T18:14:08.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.857+0000 7f679effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:08.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.857+0000 7f679effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:08.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.857+0000 7f679effd640 10 monclient(hunting): dump: 2024-03-25T18:14:08.859 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:08.859 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:08.859 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:08.859 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:08.859 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:08.859 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:08.859 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:08.859 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:08.859 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:08.859 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:08.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.857+0000 7f679effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:08.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 46943692 0 0) 0x7f67a80abb00 con 0x7f67a80a98f0 2024-03-25T18:14:08.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 >> v1:172.21.15.63:6790/0 conn(0x7f67a813f710 legacy=0x7f67a80ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 >> v1:172.21.15.81:6789/0 conn(0x7f67a81430c0 legacy=0x7f67a81e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f679effd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:08.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f679effd640 1 monclient: found mon.a 2024-03-25T18:14:08.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f679effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:08.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f679effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:14:08.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a80abde0 con 0x7f67a80a98f0 2024-03-25T18:14:08.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f679effd640 10 monclient: _finish_auth 0 2024-03-25T18:14:08.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f679effd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:08.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f679effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:08.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f67acd23640 5 monclient: authenticate success, global_id 4284 2024-03-25T18:14:08.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f67acd23640 10 monclient: _renew_subs 2024-03-25T18:14:08.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f67acd23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:14:08.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f67acd23640 1 -- 172.21.15.63:0/2004077306 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a8138f40 con 0x7f67a80a98f0 2024-03-25T18:14:08.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f67acd23640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:08.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f67acd23640 10 monclient: _renew_subs 2024-03-25T18:14:08.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f67acd23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:14:08.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f67acd23640 1 -- 172.21.15.63:0/2004077306 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a81363f0 con 0x7f67a80a98f0 2024-03-25T18:14:08.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.863+0000 7f67acd23640 1 librados: init done 2024-03-25T18:14:08.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.864+0000 7f67acd23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:08.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.864+0000 7f67acd23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:08.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.864+0000 7f67acd23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:14:08.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:08.864+0000 7f67acd23640 1 -- 172.21.15.63:0/2004077306 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a80627b0 con 0x7f67a80a98f0 2024-03-25T18:14:09.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.173+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6798001bb0 con 0x7f67a80a98f0 2024-03-25T18:14:09.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.173+0000 7f679effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:09.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.173+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6798003790 con 0x7f67a80a98f0 2024-03-25T18:14:09.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.173+0000 7f679effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.173+0000 7f679effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:09.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.173+0000 7f679effd640 10 monclient: dump: 2024-03-25T18:14:09.175 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:09.175 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:09.175 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:09.175 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:09.175 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.175 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:09.175 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:09.175 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:09.176 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:09.176 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:09.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.173+0000 7f67a559a640 4 set_mon_vals no callback set 2024-03-25T18:14:09.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.175+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f67980039b0 con 0x7f67a80a98f0 2024-03-25T18:14:09.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.175+0000 7f679ffff640 10 monclient: get_auth_request con 0x7f678806de20 auth_method 0 2024-03-25T18:14:09.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.176+0000 7f679effd640 10 client.4284.objecter ms_handle_connect 0x7f678806de20 2024-03-25T18:14:09.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.181+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f6798001d20 con 0x7f67a80a98f0 2024-03-25T18:14:09.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.181+0000 7f679effd640 10 client.4284.objecter ms_dispatch 0x7f67a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:09.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.181+0000 7f679effd640 3 client.4284.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:09.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.181+0000 7f679effd640 3 client.4284.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:09.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.181+0000 7f679effd640 20 client.4284.objecter dump_active .. 0 homeless 2024-03-25T18:14:09.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.181+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f679804ecc0 con 0x7f67a80a98f0 2024-03-25T18:14:09.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.181+0000 7f679effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:09.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.181+0000 7f679effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:09.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.443+0000 7f67acd23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:09.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.443+0000 7f67acd23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:09.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.443+0000 7f67acd23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:14:09.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.443+0000 7f67acd23640 1 -- 172.21.15.63:0/2004077306 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67a81434a0 con 0x7f67a80a98f0 2024-03-25T18:14:09.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.547+0000 7f679effd640 1 -- 172.21.15.63:0/2004077306 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f6798050dc0 con 0x7f67a80a98f0 2024-03-25T18:14:09.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.547+0000 7f679effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:09.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.547+0000 7f679effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:09.549 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:14:09.549 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:09.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.549+0000 7f67acd23640 10 librados: watch_flush enter 2024-03-25T18:14:09.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.550+0000 7f67acd23640 10 librados: watch_flush exit 2024-03-25T18:14:09.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.550+0000 7f67acd23640 20 client.4284.objecter shutdown clearing up homeless session... 2024-03-25T18:14:09.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.550+0000 7f67acd23640 10 client.4284.objecter successfully canceled tick 2024-03-25T18:14:09.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.550+0000 7f67acd23640 1 -- 172.21.15.63:0/2004077306 >> v1:172.21.15.81:6800/3640340873 conn(0x7f678806de20 legacy=0x7f67880702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.550+0000 7f67acd23640 10 monclient: shutdown 2024-03-25T18:14:09.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.550+0000 7f67acd23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:09.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.550+0000 7f67acd23640 1 -- 172.21.15.63:0/2004077306 >> v1:172.21.15.63:6789/0 conn(0x7f67a80a98f0 legacy=0x7f67a8138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.550+0000 7f67acd23640 1 -- 172.21.15.63:0/2004077306 shutdown_connections 2024-03-25T18:14:09.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.550+0000 7f67acd23640 1 -- 172.21.15.63:0/2004077306 >> 172.21.15.63:0/2004077306 conn(0x7f67a80b7ea0 msgr2=0x7f67a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:09.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.550+0000 7f67acd23640 1 -- 172.21.15.63:0/2004077306 shutdown_connections 2024-03-25T18:14:09.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.550+0000 7f67acd23640 1 -- 172.21.15.63:0/2004077306 wait complete. 2024-03-25T18:14:09.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:09.550+0000 7f67acd23640 1 librados: shutdown 2024-03-25T18:14:09.562 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:13.202 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:01:54.894 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:13.213 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:01:54.905 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:13.254 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:01:54.947 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:13.269 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:01:54.929 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:13.275 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:01:54.936 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:13.278 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:01:54.971 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:13.282 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:01:54.975 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:13.301 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:01:54.962 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:13.320 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:01:55.012 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:16.369 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:16.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.473+0000 7f594cb8e640 10 monclient: get_monmap_and_config 2024-03-25T18:14:16.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.473+0000 7f594cb8e640 10 monclient: build_initial_monmap 2024-03-25T18:14:16.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.473+0000 7f594cb8e640 10 monclient: monmap: 2024-03-25T18:14:16.474 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:14:16.474 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:16.474 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:14:16.474318+0000 2024-03-25T18:14:16.474 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:14:16.474318+0000 2024-03-25T18:14:16.474 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:16.474 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:16.475 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:14:16.475 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:14:16.475 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:14:16.475 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:16.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.473+0000 7f594cb8e640 1 Processor -- start 2024-03-25T18:14:16.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.473+0000 7f594cb8e640 1 -- start start 2024-03-25T18:14:16.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.473+0000 7f594cb8e640 10 monclient: init 2024-03-25T18:14:16.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.474+0000 7f594cb8e640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:16.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.474+0000 7f594cb8e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:16.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.474+0000 7f594cb8e640 10 monclient(hunting): picked mon.noname-b con 0x7f59480a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:14:16.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.474+0000 7f594cb8e640 10 monclient(hunting): picked mon.noname-c con 0x7f594813f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:14:16.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.474+0000 7f594cb8e640 10 monclient(hunting): picked mon.noname-a con 0x7f59481430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:14:16.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.474+0000 7f594cb8e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5948142520 con 0x7f59481430c0 2024-03-25T18:14:16.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.474+0000 7f594cb8e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59480b28a0 con 0x7f594813f710 2024-03-25T18:14:16.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.474+0000 7f594cb8e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59480b2a30 con 0x7f59480a98f0 2024-03-25T18:14:16.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.474+0000 7f594cb8e640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:16.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.474+0000 7f594cb8e640 10 monclient(hunting): authenticate will time out at 7550.661267s 2024-03-25T18:14:16.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.486+0000 7f5946d9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f59481430c0 0x7f594814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:37700/0 (socket says 172.21.15.63:37700) 2024-03-25T18:14:16.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.486+0000 7f5946d9d640 1 -- 172.21.15.63:0/2943162507 learned_addr learned my addr 172.21.15.63:0/2943162507 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:14:16.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.500+0000 7f594559a640 1 -- 172.21.15.63:0/2943162507 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172223717 0 0) 0x7f59480b2a30 con 0x7f59480a98f0 2024-03-25T18:14:16.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.500+0000 7f594559a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:16.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.500+0000 7f594559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:16.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.500+0000 7f594559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:16.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.500+0000 7f594559a640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:14:16.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.500+0000 7f594559a640 1 -- 172.21.15.63:0/2943162507 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5930003520 con 0x7f59480a98f0 2024-03-25T18:14:16.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594559a640 1 -- 172.21.15.63:0/2943162507 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3021495992 0 0) 0x7f5930003520 con 0x7f59480a98f0 2024-03-25T18:14:16.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594559a640 1 -- 172.21.15.63:0/2943162507 >> v1:172.21.15.81:6789/0 conn(0x7f594813f710 legacy=0x7f59481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:16.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594559a640 1 -- 172.21.15.63:0/2943162507 >> v1:172.21.15.63:6789/0 conn(0x7f59481430c0 legacy=0x7f594814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:16.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594559a640 10 monclient: _finish_hunting 0 2024-03-25T18:14:16.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594559a640 1 monclient: found mon.noname-b 2024-03-25T18:14:16.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:16.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:14:16.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594559a640 1 -- 172.21.15.63:0/2943162507 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59480b44c0 con 0x7f59480a98f0 2024-03-25T18:14:16.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594559a640 10 monclient: _finish_auth 0 2024-03-25T18:14:16.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594559a640 10 monclient: _check_auth_tickets 2024-03-25T18:14:16.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:16.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594559a640 1 -- 172.21.15.63:0/2943162507 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5938001690 con 0x7f59480a98f0 2024-03-25T18:14:16.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:16.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:16.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594559a640 10 monclient: dump: 2024-03-25T18:14:16.523 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:16.523 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:16.523 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:16.523 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:16.523 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:16.523 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:16.523 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:16.523 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:16.523 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:16.524 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:16.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594cb8e640 5 monclient: authenticate success, global_id 4298 2024-03-25T18:14:16.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.521+0000 7f594cb8e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:16.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.529+0000 7f594559a640 1 -- 172.21.15.63:0/2943162507 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59380017f0 con 0x7f59480a98f0 2024-03-25T18:14:16.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.529+0000 7f594559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:16.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.529+0000 7f594559a640 1 -- 172.21.15.63:0/2943162507 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5938001c40 con 0x7f59480a98f0 2024-03-25T18:14:16.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.529+0000 7f594cb8e640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:16.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.529+0000 7f594cb8e640 4 set_mon_vals no callback set 2024-03-25T18:14:16.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.529+0000 7f594559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:16.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.529+0000 7f594559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:16.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.529+0000 7f594559a640 10 monclient: dump: 2024-03-25T18:14:16.531 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:16.531 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:16.531 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:16.531 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:16.531 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:16.531 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:16.531 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:16.531 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:16.531 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:16.531 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:16.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.530+0000 7f594cb8e640 10 monclient: shutdown 2024-03-25T18:14:16.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.530+0000 7f594cb8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:16.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.530+0000 7f594cb8e640 1 -- 172.21.15.63:0/2943162507 >> v1:172.21.15.63:6790/0 conn(0x7f59480a98f0 legacy=0x7f59480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:16.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.530+0000 7f594cb8e640 1 -- 172.21.15.63:0/2943162507 shutdown_connections 2024-03-25T18:14:16.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.530+0000 7f594cb8e640 1 -- 172.21.15.63:0/2943162507 >> 172.21.15.63:0/2943162507 conn(0x7f59480b7ea0 msgr2=0x7f59480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:16.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.530+0000 7f594cb8e640 1 -- 172.21.15.63:0/2943162507 shutdown_connections 2024-03-25T18:14:16.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.530+0000 7f594cb8e640 1 -- 172.21.15.63:0/2943162507 wait complete. 2024-03-25T18:14:16.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.531+0000 7f594cb8e640 10 monclient: build_initial_monmap 2024-03-25T18:14:16.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.531+0000 7f594cb8e640 10 monclient: monmap: 2024-03-25T18:14:16.533 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:14:16.533 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:16.533 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:14:16.533 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:14:16.533 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:16.533 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:14:16.533 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:14:16.533 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:14:16.533 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:14:16.533 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:16.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.532+0000 7f594cb8e640 1 librados: starting msgr at 2024-03-25T18:14:16.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.532+0000 7f594cb8e640 1 librados: starting objecter 2024-03-25T18:14:16.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.532+0000 7f594cb8e640 1 Processor -- start 2024-03-25T18:14:16.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.532+0000 7f594cb8e640 1 -- start start 2024-03-25T18:14:16.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.532+0000 7f594cb8e640 1 librados: setting wanted keys 2024-03-25T18:14:16.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.532+0000 7f594cb8e640 1 librados: calling monclient init 2024-03-25T18:14:16.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.532+0000 7f594cb8e640 10 monclient: init 2024-03-25T18:14:16.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.533+0000 7f594cb8e640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:16.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.533+0000 7f594cb8e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:16.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.533+0000 7f594cb8e640 10 monclient(hunting): picked mon.noname-b con 0x7f59480a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:14:16.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.533+0000 7f594cb8e640 10 monclient(hunting): picked mon.noname-a con 0x7f594813f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:14:16.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.533+0000 7f594cb8e640 10 monclient(hunting): picked mon.noname-c con 0x7f59481430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:14:16.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.533+0000 7f594cb8e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5948139a40 con 0x7f594813f710 2024-03-25T18:14:16.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.533+0000 7f594cb8e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5948133f20 con 0x7f59480a98f0 2024-03-25T18:14:16.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.533+0000 7f594cb8e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59480ab950 con 0x7f59481430c0 2024-03-25T18:14:16.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.533+0000 7f594cb8e640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:16.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.533+0000 7f594cb8e640 10 monclient(hunting): authenticate will time out at 7550.719683s 2024-03-25T18:14:16.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.555+0000 7f5946d9d640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f59481430c0 0x7f5948133810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:40410/0 (socket says 172.21.15.63:40410) 2024-03-25T18:14:16.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.555+0000 7f5946d9d640 1 -- 172.21.15.63:0/2902657976 learned_addr learned my addr 172.21.15.63:0/2902657976 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:14:16.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.555+0000 7f5936ffd640 10 client.?.objecter ms_handle_connect 0x7f59481430c0 2024-03-25T18:14:16.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.555+0000 7f5936ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:16.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.562+0000 7f5936ffd640 10 client.?.objecter ms_handle_connect 0x7f59480a98f0 2024-03-25T18:14:16.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.562+0000 7f5936ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:16.564 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.563+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738307204 0 0) 0x7f59480ab950 con 0x7f59481430c0 2024-03-25T18:14:16.564 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.563+0000 7f5936ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:16.564 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.563+0000 7f5936ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:16.564 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.563+0000 7f5936ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:16.564 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.563+0000 7f5936ffd640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:14:16.564 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.563+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5928004eb0 con 0x7f59481430c0 2024-03-25T18:14:16.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.566+0000 7f5936ffd640 10 client.?.objecter ms_handle_connect 0x7f594813f710 2024-03-25T18:14:16.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.566+0000 7f5936ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:16.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.571+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129206626 0 0) 0x7f5948139a40 con 0x7f594813f710 2024-03-25T18:14:16.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.571+0000 7f5936ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:16.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.571+0000 7f5936ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:16.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.571+0000 7f5936ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:16.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.571+0000 7f5936ffd640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:14:16.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.572+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59480ab950 con 0x7f594813f710 2024-03-25T18:14:16.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.574+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2218082728 0 0) 0x7f5928004eb0 con 0x7f59481430c0 2024-03-25T18:14:16.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.574+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5948139a40 con 0x7f59481430c0 2024-03-25T18:14:16.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.574+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5940001b10 con 0x7f59481430c0 2024-03-25T18:14:16.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.574+0000 7f5936ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:16.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.574+0000 7f5936ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:16.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.574+0000 7f5936ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:16.576 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:16.576 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:16.576 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:16.576 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:16.576 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:16.576 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:16.576 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:16.576 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:16.576 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:16.576 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:16.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.574+0000 7f5936ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:16.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.583+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316880602 0 0) 0x7f5948133f20 con 0x7f59480a98f0 2024-03-25T18:14:16.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.583+0000 7f5936ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:16.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.583+0000 7f5936ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:16.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.583+0000 7f5936ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:16.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.583+0000 7f5936ffd640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:14:16.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.583+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5928004eb0 con 0x7f59480a98f0 2024-03-25T18:14:16.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3717196247 0 0) 0x7f5948139a40 con 0x7f59481430c0 2024-03-25T18:14:16.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 >> v1:172.21.15.81:6789/0 conn(0x7f59480a98f0 legacy=0x7f5948138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:16.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 >> v1:172.21.15.63:6789/0 conn(0x7f594813f710 legacy=0x7f5948133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:16.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f5936ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:16.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f5936ffd640 1 monclient: found mon.c 2024-03-25T18:14:16.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f5936ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:16.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f5936ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:14:16.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59480abc30 con 0x7f59481430c0 2024-03-25T18:14:16.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f5936ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:16.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f5936ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:16.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f5936ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:16.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f594cb8e640 5 monclient: authenticate success, global_id 4301 2024-03-25T18:14:16.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f594cb8e640 10 monclient: _renew_subs 2024-03-25T18:14:16.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f594cb8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:14:16.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f594cb8e640 1 -- 172.21.15.63:0/2902657976 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59480abf10 con 0x7f59481430c0 2024-03-25T18:14:16.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f594cb8e640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:16.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f594cb8e640 10 monclient: _renew_subs 2024-03-25T18:14:16.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f594cb8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:14:16.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f594cb8e640 1 -- 172.21.15.63:0/2902657976 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5948139370 con 0x7f59481430c0 2024-03-25T18:14:16.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.584+0000 7f594cb8e640 1 librados: init done 2024-03-25T18:14:16.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.585+0000 7f594cb8e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:16.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.585+0000 7f594cb8e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:16.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.585+0000 7f594cb8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:14:16.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.585+0000 7f594cb8e640 1 -- 172.21.15.63:0/2902657976 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59480627b0 con 0x7f59481430c0 2024-03-25T18:14:16.884 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.883+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5940002000 con 0x7f59481430c0 2024-03-25T18:14:16.884 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.883+0000 7f5936ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:16.884 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.883+0000 7f5944d99640 4 set_mon_vals no callback set 2024-03-25T18:14:16.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.894+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5940004d00 con 0x7f59481430c0 2024-03-25T18:14:16.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.894+0000 7f5936ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:16.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.894+0000 7f5936ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:16.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.894+0000 7f5936ffd640 10 monclient: dump: 2024-03-25T18:14:16.895 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:16.895 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:16.895 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:16.895 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:16.895 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:16.896 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:16.896 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:16.896 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:16.896 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:16.896 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:16.896 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.895+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f5940004ee0 con 0x7f59481430c0 2024-03-25T18:14:16.896 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.895+0000 7f594659c640 10 monclient: get_auth_request con 0x7f592806de00 auth_method 0 2024-03-25T18:14:16.896 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.896+0000 7f5936ffd640 10 client.4301.objecter ms_handle_connect 0x7f592806de00 2024-03-25T18:14:16.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.901+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f5940054aa0 con 0x7f59481430c0 2024-03-25T18:14:16.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.901+0000 7f5936ffd640 10 client.4301.objecter ms_dispatch 0x7f5948000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:16.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.901+0000 7f5936ffd640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:16.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.901+0000 7f5936ffd640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:16.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.901+0000 7f5936ffd640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-25T18:14:16.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.901+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59400503e0 con 0x7f59481430c0 2024-03-25T18:14:16.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.901+0000 7f5936ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:16.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:16.901+0000 7f5936ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.154+0000 7f594cb8e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.154+0000 7f594cb8e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.154+0000 7f594cb8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:14:17.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.154+0000 7f594cb8e640 1 -- 172.21.15.63:0/2902657976 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59480b7180 con 0x7f59481430c0 2024-03-25T18:14:17.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.201+0000 7f5936ffd640 1 -- 172.21.15.63:0/2902657976 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f59400524e0 con 0x7f59481430c0 2024-03-25T18:14:17.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.202+0000 7f5936ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:17.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.202+0000 7f5936ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:17.203 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:14:17.203 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:17.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.204+0000 7f594cb8e640 10 librados: watch_flush enter 2024-03-25T18:14:17.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.204+0000 7f594cb8e640 10 librados: watch_flush exit 2024-03-25T18:14:17.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.204+0000 7f594cb8e640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-25T18:14:17.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.204+0000 7f594cb8e640 10 client.4301.objecter successfully canceled tick 2024-03-25T18:14:17.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.204+0000 7f594cb8e640 1 -- 172.21.15.63:0/2902657976 >> v1:172.21.15.81:6800/3640340873 conn(0x7f592806de00 legacy=0x7f59280702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.204+0000 7f594cb8e640 10 monclient: shutdown 2024-03-25T18:14:17.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.204+0000 7f594cb8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:17.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.204+0000 7f594cb8e640 1 -- 172.21.15.63:0/2902657976 >> v1:172.21.15.63:6790/0 conn(0x7f59481430c0 legacy=0x7f5948133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.204+0000 7f594cb8e640 1 -- 172.21.15.63:0/2902657976 shutdown_connections 2024-03-25T18:14:17.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.204+0000 7f594cb8e640 1 -- 172.21.15.63:0/2902657976 >> 172.21.15.63:0/2902657976 conn(0x7f59480b7ea0 msgr2=0x7f5948074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:17.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.204+0000 7f594cb8e640 1 -- 172.21.15.63:0/2902657976 shutdown_connections 2024-03-25T18:14:17.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.204+0000 7f594cb8e640 1 -- 172.21.15.63:0/2902657976 wait complete. 2024-03-25T18:14:17.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:17.204+0000 7f594cb8e640 1 librados: shutdown 2024-03-25T18:14:17.217 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:24.024 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:24.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.143+0000 7fda2ca63640 10 monclient: get_monmap_and_config 2024-03-25T18:14:24.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.143+0000 7fda2ca63640 10 monclient: build_initial_monmap 2024-03-25T18:14:24.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.143+0000 7fda2ca63640 10 monclient: monmap: 2024-03-25T18:14:24.152 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:14:24.152 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:24.152 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:14:24.144364+0000 2024-03-25T18:14:24.152 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:14:24.144364+0000 2024-03-25T18:14:24.152 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:24.152 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:24.152 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:14:24.152 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:14:24.152 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:14:24.152 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:24.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.143+0000 7fda2ca63640 1 Processor -- start 2024-03-25T18:14:24.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.143+0000 7fda2ca63640 1 -- start start 2024-03-25T18:14:24.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.144+0000 7fda2ca63640 10 monclient: init 2024-03-25T18:14:24.153 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.144+0000 7fda2ca63640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:24.153 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.144+0000 7fda2ca63640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:24.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.144+0000 7fda2ca63640 10 monclient(hunting): picked mon.noname-c con 0x7fda280a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:14:24.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.144+0000 7fda2ca63640 10 monclient(hunting): picked mon.noname-b con 0x7fda2813f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:14:24.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.144+0000 7fda2ca63640 10 monclient(hunting): picked mon.noname-a con 0x7fda281430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:14:24.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.144+0000 7fda2ca63640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda28142520 con 0x7fda281430c0 2024-03-25T18:14:24.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.144+0000 7fda2ca63640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda280b28a0 con 0x7fda280a98f0 2024-03-25T18:14:24.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.144+0000 7fda2ca63640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda280b2a30 con 0x7fda2813f710 2024-03-25T18:14:24.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.144+0000 7fda2ca63640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:24.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.144+0000 7fda2ca63640 10 monclient(hunting): authenticate will time out at 7558.331199s 2024-03-25T18:14:24.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.148+0000 7fda2659c640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fda280a98f0 0x7fda280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:43928/0 (socket says 172.21.15.63:43928) 2024-03-25T18:14:24.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.148+0000 7fda2659c640 1 -- 172.21.15.63:0/2913264611 learned_addr learned my addr 172.21.15.63:0/2913264611 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:14:24.164 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.163+0000 7fda2559a640 1 -- 172.21.15.63:0/2913264611 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404722678 0 0) 0x7fda280b28a0 con 0x7fda280a98f0 2024-03-25T18:14:24.164 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.164+0000 7fda2559a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:24.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.164+0000 7fda2559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:24.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.164+0000 7fda2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:24.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.164+0000 7fda2559a640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:14:24.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.164+0000 7fda2559a640 1 -- 172.21.15.63:0/2913264611 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda10003520 con 0x7fda280a98f0 2024-03-25T18:14:24.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.177+0000 7fda2559a640 1 -- 172.21.15.63:0/2913264611 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893942794 0 0) 0x7fda28142520 con 0x7fda281430c0 2024-03-25T18:14:24.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.177+0000 7fda2559a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:24.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.177+0000 7fda2559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:24.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.177+0000 7fda2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:24.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.177+0000 7fda2559a640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:14:24.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.177+0000 7fda2559a640 1 -- 172.21.15.63:0/2913264611 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda280b28a0 con 0x7fda281430c0 2024-03-25T18:14:24.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2559a640 1 -- 172.21.15.63:0/2913264611 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1479613716 0 0) 0x7fda10003520 con 0x7fda280a98f0 2024-03-25T18:14:24.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2559a640 1 -- 172.21.15.63:0/2913264611 >> v1:172.21.15.63:6790/0 conn(0x7fda2813f710 legacy=0x7fda281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:14:24.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2559a640 1 -- 172.21.15.63:0/2913264611 >> v1:172.21.15.63:6789/0 conn(0x7fda281430c0 legacy=0x7fda2814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:24.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2559a640 10 monclient: _finish_hunting 0 2024-03-25T18:14:24.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2559a640 1 monclient: found mon.noname-c 2024-03-25T18:14:24.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:24.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:14:24.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2559a640 1 -- 172.21.15.63:0/2913264611 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda280b44c0 con 0x7fda280a98f0 2024-03-25T18:14:24.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2559a640 10 monclient: _finish_auth 0 2024-03-25T18:14:24.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2559a640 10 monclient: _check_auth_tickets 2024-03-25T18:14:24.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:24.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2559a640 1 -- 172.21.15.63:0/2913264611 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fda18001690 con 0x7fda280a98f0 2024-03-25T18:14:24.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:24.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:24.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2559a640 10 monclient: dump: 2024-03-25T18:14:24.184 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:24.184 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:24.184 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:24.184 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:24.184 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:24.184 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:24.184 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:24.184 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:24.185 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:24.185 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:24.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2ca63640 5 monclient: authenticate success, global_id 4303 2024-03-25T18:14:24.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.181+0000 7fda2ca63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:24.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.203+0000 7fda2559a640 1 -- 172.21.15.63:0/2913264611 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda180017f0 con 0x7fda280a98f0 2024-03-25T18:14:24.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.203+0000 7fda2559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:24.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.203+0000 7fda2559a640 1 -- 172.21.15.63:0/2913264611 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fda18001c00 con 0x7fda280a98f0 2024-03-25T18:14:24.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.203+0000 7fda2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:24.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.203+0000 7fda2559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:24.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.203+0000 7fda2559a640 10 monclient: dump: 2024-03-25T18:14:24.205 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:24.205 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:24.205 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:24.205 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:24.205 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:24.205 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:24.205 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:24.205 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:24.205 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:24.205 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:24.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.203+0000 7fda2ca63640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:24.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.203+0000 7fda2ca63640 4 set_mon_vals no callback set 2024-03-25T18:14:24.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.204+0000 7fda2ca63640 10 monclient: shutdown 2024-03-25T18:14:24.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.204+0000 7fda2ca63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:24.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.204+0000 7fda2ca63640 1 -- 172.21.15.63:0/2913264611 >> v1:172.21.15.81:6789/0 conn(0x7fda280a98f0 legacy=0x7fda280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:24.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.204+0000 7fda2ca63640 1 -- 172.21.15.63:0/2913264611 shutdown_connections 2024-03-25T18:14:24.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.204+0000 7fda2ca63640 1 -- 172.21.15.63:0/2913264611 >> 172.21.15.63:0/2913264611 conn(0x7fda280b7ea0 msgr2=0x7fda280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:24.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.204+0000 7fda2ca63640 1 -- 172.21.15.63:0/2913264611 shutdown_connections 2024-03-25T18:14:24.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.204+0000 7fda2ca63640 1 -- 172.21.15.63:0/2913264611 wait complete. 2024-03-25T18:14:24.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.205+0000 7fda2ca63640 10 monclient: build_initial_monmap 2024-03-25T18:14:24.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.205+0000 7fda2ca63640 10 monclient: monmap: 2024-03-25T18:14:24.209 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:14:24.209 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:24.209 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:14:24.209 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:14:24.209 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:24.209 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:14:24.209 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:14:24.209 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:14:24.209 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:14:24.209 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:24.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.205+0000 7fda2ca63640 1 librados: starting msgr at 2024-03-25T18:14:24.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.205+0000 7fda2ca63640 1 librados: starting objecter 2024-03-25T18:14:24.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.206+0000 7fda2ca63640 1 Processor -- start 2024-03-25T18:14:24.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.206+0000 7fda2ca63640 1 -- start start 2024-03-25T18:14:24.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.206+0000 7fda2ca63640 1 librados: setting wanted keys 2024-03-25T18:14:24.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.206+0000 7fda2ca63640 1 librados: calling monclient init 2024-03-25T18:14:24.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.206+0000 7fda2ca63640 10 monclient: init 2024-03-25T18:14:24.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.206+0000 7fda2ca63640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:24.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.206+0000 7fda2ca63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:24.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.206+0000 7fda2ca63640 10 monclient(hunting): picked mon.noname-c con 0x7fda280a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:14:24.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.206+0000 7fda2ca63640 10 monclient(hunting): picked mon.noname-a con 0x7fda2813f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:14:24.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.206+0000 7fda2ca63640 10 monclient(hunting): picked mon.noname-b con 0x7fda281430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:14:24.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.206+0000 7fda2ca63640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda280ac1f0 con 0x7fda2813f710 2024-03-25T18:14:24.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.206+0000 7fda2ca63640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda28137f90 con 0x7fda281430c0 2024-03-25T18:14:24.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.206+0000 7fda2ca63640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda28139330 con 0x7fda280a98f0 2024-03-25T18:14:24.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.206+0000 7fda2ca63640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:24.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.206+0000 7fda2ca63640 10 monclient(hunting): authenticate will time out at 7558.393058s 2024-03-25T18:14:24.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.215+0000 7fda26d9d640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fda281430c0 0x7fda28138c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:43968/0 (socket says 172.21.15.63:43968) 2024-03-25T18:14:24.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.215+0000 7fda26d9d640 1 -- 172.21.15.63:0/3173959195 learned_addr learned my addr 172.21.15.63:0/3173959195 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:14:24.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.218+0000 7fda16ffd640 10 client.?.objecter ms_handle_connect 0x7fda281430c0 2024-03-25T18:14:24.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.218+0000 7fda16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:24.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.229+0000 7fda16ffd640 10 client.?.objecter ms_handle_connect 0x7fda2813f710 2024-03-25T18:14:24.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.229+0000 7fda16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:24.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.229+0000 7fda16ffd640 10 client.?.objecter ms_handle_connect 0x7fda280a98f0 2024-03-25T18:14:24.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.229+0000 7fda16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:24.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.238+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399377429 0 0) 0x7fda28137f90 con 0x7fda281430c0 2024-03-25T18:14:24.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.238+0000 7fda16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:24.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.238+0000 7fda16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:24.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.238+0000 7fda16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:24.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.238+0000 7fda16ffd640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:14:24.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.238+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda08004eb0 con 0x7fda281430c0 2024-03-25T18:14:24.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.242+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616771783 0 0) 0x7fda280ac1f0 con 0x7fda2813f710 2024-03-25T18:14:24.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.242+0000 7fda16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:24.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.242+0000 7fda16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:24.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.242+0000 7fda16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:24.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.242+0000 7fda16ffd640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:14:24.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.242+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda28137f90 con 0x7fda2813f710 2024-03-25T18:14:24.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.244+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409176074 0 0) 0x7fda28139330 con 0x7fda280a98f0 2024-03-25T18:14:24.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.244+0000 7fda16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:24.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.244+0000 7fda16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:24.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.244+0000 7fda16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:24.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.244+0000 7fda16ffd640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:14:24.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.244+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda280ac1f0 con 0x7fda280a98f0 2024-03-25T18:14:24.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.249+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2688329174 0 0) 0x7fda08004eb0 con 0x7fda281430c0 2024-03-25T18:14:24.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.250+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda28139330 con 0x7fda281430c0 2024-03-25T18:14:24.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.250+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fda20001c70 con 0x7fda281430c0 2024-03-25T18:14:24.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.250+0000 7fda16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:24.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.250+0000 7fda16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:24.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.250+0000 7fda16ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:24.251 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:24.251 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:24.251 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:24.251 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:24.251 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:24.251 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:24.251 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:24.251 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:24.251 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:24.251 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:24.252 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.250+0000 7fda16ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:24.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.257+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1360609888 0 0) 0x7fda28137f90 con 0x7fda2813f710 2024-03-25T18:14:24.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.257+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda08004eb0 con 0x7fda2813f710 2024-03-25T18:14:24.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.257+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fda1c001510 con 0x7fda2813f710 2024-03-25T18:14:24.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.257+0000 7fda16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:24.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.257+0000 7fda16ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:24.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.257+0000 7fda16ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:24.258 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:24.258 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:24.258 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:24.258 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:24.258 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:24.258 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:24.259 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:24.259 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:24.259 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:24.259 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:24.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2730203706 0 0) 0x7fda28139330 con 0x7fda281430c0 2024-03-25T18:14:24.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 >> v1:172.21.15.63:6790/0 conn(0x7fda280a98f0 legacy=0x7fda280ac610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:24.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 >> v1:172.21.15.63:6789/0 conn(0x7fda2813f710 legacy=0x7fda281365e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:24.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda16ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:24.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda16ffd640 1 monclient: found mon.b 2024-03-25T18:14:24.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:24.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda16ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:24.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda281394c0 con 0x7fda281430c0 2024-03-25T18:14:24.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda16ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:24.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda16ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:24.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:24.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda2ca63640 5 monclient: authenticate success, global_id 4306 2024-03-25T18:14:24.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda2ca63640 10 monclient: _renew_subs 2024-03-25T18:14:24.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda2ca63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:24.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda2ca63640 1 -- 172.21.15.63:0/3173959195 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda28139880 con 0x7fda281430c0 2024-03-25T18:14:24.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda2ca63640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:24.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda2ca63640 10 monclient: _renew_subs 2024-03-25T18:14:24.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda2ca63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:24.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda2ca63640 1 -- 172.21.15.63:0/3173959195 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda281e38a0 con 0x7fda281430c0 2024-03-25T18:14:24.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.271+0000 7fda2ca63640 1 librados: init done 2024-03-25T18:14:24.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.273+0000 7fda2ca63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:24.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.273+0000 7fda2ca63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:24.275 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.273+0000 7fda2ca63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:24.275 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.273+0000 7fda2ca63640 1 -- 172.21.15.63:0/3173959195 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda281381d0 con 0x7fda281430c0 2024-03-25T18:14:24.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.556+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda20003990 con 0x7fda281430c0 2024-03-25T18:14:24.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.556+0000 7fda16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:24.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.556+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fda20003b40 con 0x7fda281430c0 2024-03-25T18:14:24.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.556+0000 7fda16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:24.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.556+0000 7fda16ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:24.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.556+0000 7fda16ffd640 10 monclient: dump: 2024-03-25T18:14:24.558 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:24.558 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:24.558 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:24.558 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:24.558 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:24.558 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:24.558 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:24.558 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:24.558 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:24.558 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:24.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.556+0000 7fda24d99640 4 set_mon_vals no callback set 2024-03-25T18:14:24.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.557+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fda20003d60 con 0x7fda281430c0 2024-03-25T18:14:24.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.558+0000 7fda2659c640 10 monclient: get_auth_request con 0x7fda0806df70 auth_method 0 2024-03-25T18:14:24.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.558+0000 7fda16ffd640 10 client.4306.objecter ms_handle_connect 0x7fda0806df70 2024-03-25T18:14:24.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.562+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fda20054980 con 0x7fda281430c0 2024-03-25T18:14:24.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.562+0000 7fda16ffd640 10 client.4306.objecter ms_dispatch 0x7fda28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:24.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.562+0000 7fda16ffd640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:24.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.562+0000 7fda16ffd640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:24.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.562+0000 7fda16ffd640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-25T18:14:24.564 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.563+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fda20050490 con 0x7fda281430c0 2024-03-25T18:14:24.564 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.563+0000 7fda16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:24.564 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.563+0000 7fda16ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:24.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.837+0000 7fda2ca63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:24.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.837+0000 7fda2ca63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:24.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.837+0000 7fda2ca63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:24.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.837+0000 7fda2ca63640 1 -- 172.21.15.63:0/3173959195 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda28136cf0 con 0x7fda281430c0 2024-03-25T18:14:24.875 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.874+0000 7fda16ffd640 1 -- 172.21.15.63:0/3173959195 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fda28136cf0 con 0x7fda281430c0 2024-03-25T18:14:24.875 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.874+0000 7fda16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:24.875 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.874+0000 7fda16ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:24.875 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:14:24.875 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:24.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.876+0000 7fda2ca63640 10 librados: watch_flush enter 2024-03-25T18:14:24.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.877+0000 7fda2ca63640 10 librados: watch_flush exit 2024-03-25T18:14:24.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.877+0000 7fda2ca63640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-25T18:14:24.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.877+0000 7fda2ca63640 10 client.4306.objecter successfully canceled tick 2024-03-25T18:14:24.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.877+0000 7fda2ca63640 1 -- 172.21.15.63:0/3173959195 >> v1:172.21.15.81:6800/3640340873 conn(0x7fda0806df70 legacy=0x7fda08070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:24.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.877+0000 7fda2ca63640 10 monclient: shutdown 2024-03-25T18:14:24.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.877+0000 7fda2ca63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:24.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.877+0000 7fda2ca63640 1 -- 172.21.15.63:0/3173959195 >> v1:172.21.15.81:6789/0 conn(0x7fda281430c0 legacy=0x7fda28138c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:24.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.877+0000 7fda2ca63640 1 -- 172.21.15.63:0/3173959195 shutdown_connections 2024-03-25T18:14:24.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.877+0000 7fda2ca63640 1 -- 172.21.15.63:0/3173959195 >> 172.21.15.63:0/3173959195 conn(0x7fda280b7ea0 msgr2=0x7fda28074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:24.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.877+0000 7fda2ca63640 1 -- 172.21.15.63:0/3173959195 shutdown_connections 2024-03-25T18:14:24.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.877+0000 7fda2ca63640 1 -- 172.21.15.63:0/3173959195 wait complete. 2024-03-25T18:14:24.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:24.877+0000 7fda2ca63640 1 librados: shutdown 2024-03-25T18:14:24.889 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:31.695 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:31.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.812+0000 7f1a7b38c640 10 monclient: get_monmap_and_config 2024-03-25T18:14:31.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.812+0000 7f1a7b38c640 10 monclient: build_initial_monmap 2024-03-25T18:14:31.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.812+0000 7f1a7b38c640 10 monclient: monmap: 2024-03-25T18:14:31.813 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:14:31.813 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:31.813 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:14:31.813584+0000 2024-03-25T18:14:31.813 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:14:31.813584+0000 2024-03-25T18:14:31.813 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:31.813 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:31.814 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:14:31.814 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:14:31.814 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:14:31.814 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:31.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.813+0000 7f1a7b38c640 1 Processor -- start 2024-03-25T18:14:31.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.813+0000 7f1a7b38c640 1 -- start start 2024-03-25T18:14:31.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.813+0000 7f1a7b38c640 10 monclient: init 2024-03-25T18:14:31.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.813+0000 7f1a7b38c640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:31.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.813+0000 7f1a7b38c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:31.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.813+0000 7f1a7b38c640 10 monclient(hunting): picked mon.noname-c con 0x7f1a740a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:14:31.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.813+0000 7f1a7b38c640 10 monclient(hunting): picked mon.noname-b con 0x7f1a7413f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:14:31.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.813+0000 7f1a7b38c640 10 monclient(hunting): picked mon.noname-a con 0x7f1a741430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:14:31.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.813+0000 7f1a7b38c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a74142520 con 0x7f1a741430c0 2024-03-25T18:14:31.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.813+0000 7f1a7b38c640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a740b28a0 con 0x7f1a740a98f0 2024-03-25T18:14:31.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.813+0000 7f1a7b38c640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a740b2a30 con 0x7f1a7413f710 2024-03-25T18:14:31.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.813+0000 7f1a7b38c640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:31.816 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.813+0000 7f1a7b38c640 10 monclient(hunting): authenticate will time out at 7566.000234s 2024-03-25T18:14:31.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.826+0000 7f1a79128640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f1a740a98f0 0x7f1a740a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:39468/0 (socket says 172.21.15.63:39468) 2024-03-25T18:14:31.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.826+0000 7f1a79128640 1 -- 172.21.15.63:0/4223532487 learned_addr learned my addr 172.21.15.63:0/4223532487 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:14:31.837 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.836+0000 7f1a6bfff640 1 -- 172.21.15.63:0/4223532487 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609709131 0 0) 0x7f1a740b28a0 con 0x7f1a740a98f0 2024-03-25T18:14:31.837 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.836+0000 7f1a6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:31.837 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.836+0000 7f1a6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:31.837 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.836+0000 7f1a6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:31.837 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.836+0000 7f1a6bfff640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:14:31.837 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.836+0000 7f1a6bfff640 1 -- 172.21.15.63:0/4223532487 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a60003520 con 0x7f1a740a98f0 2024-03-25T18:14:31.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.848+0000 7f1a6bfff640 1 -- 172.21.15.63:0/4223532487 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838175350 0 0) 0x7f1a740b2a30 con 0x7f1a7413f710 2024-03-25T18:14:31.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.848+0000 7f1a6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:31.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.848+0000 7f1a6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:31.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.848+0000 7f1a6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:31.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.848+0000 7f1a6bfff640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:14:31.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.848+0000 7f1a6bfff640 1 -- 172.21.15.63:0/4223532487 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a740b28a0 con 0x7f1a7413f710 2024-03-25T18:14:31.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.848+0000 7f1a6bfff640 1 -- 172.21.15.63:0/4223532487 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529482189 0 0) 0x7f1a74142520 con 0x7f1a741430c0 2024-03-25T18:14:31.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.848+0000 7f1a6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:31.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.848+0000 7f1a6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:31.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.848+0000 7f1a6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:31.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.848+0000 7f1a6bfff640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:14:31.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.848+0000 7f1a6bfff640 1 -- 172.21.15.63:0/4223532487 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a740b2a30 con 0x7f1a741430c0 2024-03-25T18:14:31.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.856+0000 7f1a6bfff640 1 -- 172.21.15.63:0/4223532487 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1196128422 0 0) 0x7f1a60003520 con 0x7f1a740a98f0 2024-03-25T18:14:31.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.856+0000 7f1a6bfff640 1 -- 172.21.15.63:0/4223532487 >> v1:172.21.15.63:6790/0 conn(0x7f1a7413f710 legacy=0x7f1a741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:31.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.857+0000 7f1a6bfff640 1 -- 172.21.15.63:0/4223532487 >> v1:172.21.15.63:6789/0 conn(0x7f1a741430c0 legacy=0x7f1a7414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:31.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.857+0000 7f1a6bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:31.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.857+0000 7f1a6bfff640 1 monclient: found mon.noname-c 2024-03-25T18:14:31.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.857+0000 7f1a6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:31.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.857+0000 7f1a6bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:14:31.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.857+0000 7f1a6bfff640 1 -- 172.21.15.63:0/4223532487 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a740b44c0 con 0x7f1a740a98f0 2024-03-25T18:14:31.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.857+0000 7f1a6bfff640 10 monclient: _finish_auth 0 2024-03-25T18:14:31.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.857+0000 7f1a6bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:31.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.857+0000 7f1a6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:31.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.857+0000 7f1a6bfff640 1 -- 172.21.15.63:0/4223532487 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f1a70001690 con 0x7f1a740a98f0 2024-03-25T18:14:31.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.857+0000 7f1a6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:31.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.857+0000 7f1a6bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:31.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.857+0000 7f1a6bfff640 10 monclient: dump: 2024-03-25T18:14:31.859 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:31.859 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:31.859 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:31.859 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:31.859 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:31.859 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:31.859 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:31.860 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:31.860 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:31.860 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:31.860 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.857+0000 7f1a7b38c640 5 monclient: authenticate success, global_id 4309 2024-03-25T18:14:31.860 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.857+0000 7f1a7b38c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:31.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.890+0000 7f1a6bfff640 1 -- 172.21.15.63:0/4223532487 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a700017f0 con 0x7f1a740a98f0 2024-03-25T18:14:31.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.890+0000 7f1a6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:31.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.890+0000 7f1a6bfff640 1 -- 172.21.15.63:0/4223532487 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f1a70001c00 con 0x7f1a740a98f0 2024-03-25T18:14:31.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.890+0000 7f1a7b38c640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:31.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.890+0000 7f1a7b38c640 4 set_mon_vals no callback set 2024-03-25T18:14:31.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.890+0000 7f1a6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:31.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.890+0000 7f1a6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:31.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.890+0000 7f1a6bfff640 10 monclient: dump: 2024-03-25T18:14:31.892 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:31.892 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:31.892 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:31.892 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:31.892 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:31.892 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:31.892 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:31.892 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:31.892 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:31.892 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:31.893 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.891+0000 7f1a7b38c640 10 monclient: shutdown 2024-03-25T18:14:31.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.891+0000 7f1a7b38c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:31.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.891+0000 7f1a7b38c640 1 -- 172.21.15.63:0/4223532487 >> v1:172.21.15.81:6789/0 conn(0x7f1a740a98f0 legacy=0x7f1a740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:31.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.891+0000 7f1a7b38c640 1 -- 172.21.15.63:0/4223532487 shutdown_connections 2024-03-25T18:14:31.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.891+0000 7f1a7b38c640 1 -- 172.21.15.63:0/4223532487 >> 172.21.15.63:0/4223532487 conn(0x7f1a740b7ea0 msgr2=0x7f1a740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:31.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.891+0000 7f1a7b38c640 1 -- 172.21.15.63:0/4223532487 shutdown_connections 2024-03-25T18:14:31.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.891+0000 7f1a7b38c640 1 -- 172.21.15.63:0/4223532487 wait complete. 2024-03-25T18:14:31.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.892+0000 7f1a7b38c640 10 monclient: build_initial_monmap 2024-03-25T18:14:31.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.892+0000 7f1a7b38c640 10 monclient: monmap: 2024-03-25T18:14:31.894 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:14:31.894 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:31.894 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:14:31.895 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:14:31.895 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:31.895 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:14:31.895 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:14:31.895 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:14:31.895 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:14:31.895 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:31.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.893+0000 7f1a7b38c640 1 librados: starting msgr at 2024-03-25T18:14:31.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.893+0000 7f1a7b38c640 1 librados: starting objecter 2024-03-25T18:14:31.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.893+0000 7f1a7b38c640 1 Processor -- start 2024-03-25T18:14:31.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.893+0000 7f1a7b38c640 1 -- start start 2024-03-25T18:14:31.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.893+0000 7f1a7b38c640 1 librados: setting wanted keys 2024-03-25T18:14:31.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.893+0000 7f1a7b38c640 1 librados: calling monclient init 2024-03-25T18:14:31.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.893+0000 7f1a7b38c640 10 monclient: init 2024-03-25T18:14:31.896 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.894+0000 7f1a7b38c640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:31.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.894+0000 7f1a7b38c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:31.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.894+0000 7f1a7b38c640 10 monclient(hunting): picked mon.noname-c con 0x7f1a740a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:14:31.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.894+0000 7f1a7b38c640 10 monclient(hunting): picked mon.noname-b con 0x7f1a7413f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:14:31.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.894+0000 7f1a7b38c640 10 monclient(hunting): picked mon.noname-a con 0x7f1a741430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:14:31.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.894+0000 7f1a7b38c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a740abfc0 con 0x7f1a741430c0 2024-03-25T18:14:31.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.894+0000 7f1a7b38c640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a741dc150 con 0x7f1a7413f710 2024-03-25T18:14:31.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.894+0000 7f1a7b38c640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a74137980 con 0x7f1a740a98f0 2024-03-25T18:14:31.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.894+0000 7f1a7b38c640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:31.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.894+0000 7f1a7b38c640 10 monclient(hunting): authenticate will time out at 7566.080726s 2024-03-25T18:14:31.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.897+0000 7f1a79929640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f1a741430c0 0x7f1a741dba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:51400/0 (socket says 172.21.15.63:51400) 2024-03-25T18:14:31.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.897+0000 7f1a79929640 1 -- 172.21.15.63:0/2950661901 learned_addr learned my addr 172.21.15.63:0/2950661901 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:14:31.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.901+0000 7f1a69ffb640 10 client.?.objecter ms_handle_connect 0x7f1a740a98f0 2024-03-25T18:14:31.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.901+0000 7f1a69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:31.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.902+0000 7f1a69ffb640 10 client.?.objecter ms_handle_connect 0x7f1a741430c0 2024-03-25T18:14:31.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.902+0000 7f1a69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:31.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.910+0000 7f1a69ffb640 10 client.?.objecter ms_handle_connect 0x7f1a7413f710 2024-03-25T18:14:31.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.910+0000 7f1a69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:31.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.912+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44295561 0 0) 0x7f1a74137980 con 0x7f1a740a98f0 2024-03-25T18:14:31.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.912+0000 7f1a69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:31.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.912+0000 7f1a69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:31.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.912+0000 7f1a69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:31.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.912+0000 7f1a69ffb640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:14:31.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.912+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a58004eb0 con 0x7f1a740a98f0 2024-03-25T18:14:31.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.916+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116537180 0 0) 0x7f1a741dc150 con 0x7f1a7413f710 2024-03-25T18:14:31.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.916+0000 7f1a69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:31.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.916+0000 7f1a69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:31.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.916+0000 7f1a69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:31.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.916+0000 7f1a69ffb640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:14:31.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.916+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a74137980 con 0x7f1a7413f710 2024-03-25T18:14:31.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.923+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284157392 0 0) 0x7f1a740abfc0 con 0x7f1a741430c0 2024-03-25T18:14:31.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.923+0000 7f1a69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:31.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.923+0000 7f1a69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:31.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.923+0000 7f1a69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:31.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.923+0000 7f1a69ffb640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:14:31.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.923+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a741dc150 con 0x7f1a741430c0 2024-03-25T18:14:31.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.946+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4061932195 0 0) 0x7f1a74137980 con 0x7f1a7413f710 2024-03-25T18:14:31.947 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.946+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a740abfc0 con 0x7f1a7413f710 2024-03-25T18:14:31.947 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.946+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f1a64001970 con 0x7f1a7413f710 2024-03-25T18:14:31.947 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.946+0000 7f1a69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:31.947 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.946+0000 7f1a69ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:31.947 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.946+0000 7f1a69ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:31.947 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:31.947 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:31.947 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:31.947 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:31.947 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:31.947 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:31.947 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:31.947 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:31.948 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:31.948 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:31.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.946+0000 7f1a69ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:31.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.947+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3672737536 0 0) 0x7f1a58004eb0 con 0x7f1a740a98f0 2024-03-25T18:14:31.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.947+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a74137980 con 0x7f1a740a98f0 2024-03-25T18:14:31.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.947+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f1a70001b00 con 0x7f1a740a98f0 2024-03-25T18:14:31.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.947+0000 7f1a69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:31.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.947+0000 7f1a69ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:31.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.947+0000 7f1a69ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:31.949 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:31.949 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:31.949 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:31.949 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:31.949 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:31.949 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:31.949 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:31.949 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:31.949 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:31.949 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:31.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.963+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2187666220 0 0) 0x7f1a740abfc0 con 0x7f1a7413f710 2024-03-25T18:14:31.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 >> v1:172.21.15.63:6790/0 conn(0x7f1a740a98f0 legacy=0x7f1a740ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:31.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 >> v1:172.21.15.63:6789/0 conn(0x7f1a741430c0 legacy=0x7f1a741dba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:31.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a69ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:31.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a69ffb640 1 monclient: found mon.b 2024-03-25T18:14:31.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:31.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a69ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:31.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a74139360 con 0x7f1a7413f710 2024-03-25T18:14:31.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a69ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:31.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a69ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:31.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:31.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a7b38c640 5 monclient: authenticate success, global_id 4312 2024-03-25T18:14:31.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a7b38c640 10 monclient: _renew_subs 2024-03-25T18:14:31.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a7b38c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:31.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a7b38c640 1 -- 172.21.15.63:0/2950661901 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a74139640 con 0x7f1a7413f710 2024-03-25T18:14:31.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a7b38c640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:31.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a7b38c640 10 monclient: _renew_subs 2024-03-25T18:14:31.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a7b38c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:31.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a7b38c640 1 -- 172.21.15.63:0/2950661901 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a74136f70 con 0x7f1a7413f710 2024-03-25T18:14:31.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.964+0000 7f1a7b38c640 1 librados: init done 2024-03-25T18:14:31.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.965+0000 7f1a7b38c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:31.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.965+0000 7f1a7b38c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:31.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.965+0000 7f1a7b38c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:31.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:31.965+0000 7f1a7b38c640 1 -- 172.21.15.63:0/2950661901 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a74137b10 con 0x7f1a7413f710 2024-03-25T18:14:32.231 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.230+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a64003610 con 0x7f1a7413f710 2024-03-25T18:14:32.231 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.230+0000 7f1a69ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:32.231 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.230+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f1a64003920 con 0x7f1a7413f710 2024-03-25T18:14:32.231 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.230+0000 7f1a69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:32.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.230+0000 7f1a69ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:32.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.230+0000 7f1a69ffb640 10 monclient: dump: 2024-03-25T18:14:32.232 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:32.232 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:32.232 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:32.232 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:32.232 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:32.232 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:32.232 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:32.232 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:32.232 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:32.232 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:32.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.230+0000 7f1a6b7fe640 4 set_mon_vals no callback set 2024-03-25T18:14:32.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.231+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f1a640198f0 con 0x7f1a7413f710 2024-03-25T18:14:32.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.231+0000 7f1a79128640 10 monclient: get_auth_request con 0x7f1a5806df70 auth_method 0 2024-03-25T18:14:32.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.232+0000 7f1a69ffb640 10 client.4312.objecter ms_handle_connect 0x7f1a5806df70 2024-03-25T18:14:32.237 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.236+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f1a64053580 con 0x7f1a7413f710 2024-03-25T18:14:32.237 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.236+0000 7f1a69ffb640 10 client.4312.objecter ms_dispatch 0x7f1a74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:32.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.236+0000 7f1a69ffb640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:32.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.236+0000 7f1a69ffb640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:32.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.236+0000 7f1a69ffb640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-25T18:14:32.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.236+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a6404eec0 con 0x7f1a7413f710 2024-03-25T18:14:32.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.237+0000 7f1a69ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:32.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.237+0000 7f1a69ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:32.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.469+0000 7f1a7b38c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:32.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.469+0000 7f1a7b38c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:32.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.469+0000 7f1a7b38c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:32.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.469+0000 7f1a7b38c640 1 -- 172.21.15.63:0/2950661901 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a74137270 con 0x7f1a7413f710 2024-03-25T18:14:32.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.520+0000 7f1a69ffb640 1 -- 172.21.15.63:0/2950661901 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f1a64050fc0 con 0x7f1a7413f710 2024-03-25T18:14:32.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.520+0000 7f1a69ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:32.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.520+0000 7f1a69ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:32.522 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:14:32.522 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:32.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.523+0000 7f1a7b38c640 10 librados: watch_flush enter 2024-03-25T18:14:32.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.523+0000 7f1a7b38c640 10 librados: watch_flush exit 2024-03-25T18:14:32.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.523+0000 7f1a7b38c640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-25T18:14:32.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.523+0000 7f1a7b38c640 10 client.4312.objecter successfully canceled tick 2024-03-25T18:14:32.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.523+0000 7f1a7b38c640 1 -- 172.21.15.63:0/2950661901 >> v1:172.21.15.81:6800/3640340873 conn(0x7f1a5806df70 legacy=0x7f1a58070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:32.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.523+0000 7f1a7b38c640 10 monclient: shutdown 2024-03-25T18:14:32.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.523+0000 7f1a7b38c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:32.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.523+0000 7f1a7b38c640 1 -- 172.21.15.63:0/2950661901 >> v1:172.21.15.81:6789/0 conn(0x7f1a7413f710 legacy=0x7f1a741db330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:32.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.523+0000 7f1a7b38c640 1 -- 172.21.15.63:0/2950661901 shutdown_connections 2024-03-25T18:14:32.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.523+0000 7f1a7b38c640 1 -- 172.21.15.63:0/2950661901 >> 172.21.15.63:0/2950661901 conn(0x7f1a740b7ea0 msgr2=0x7f1a74074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:32.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.523+0000 7f1a7b38c640 1 -- 172.21.15.63:0/2950661901 shutdown_connections 2024-03-25T18:14:32.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.523+0000 7f1a7b38c640 1 -- 172.21.15.63:0/2950661901 wait complete. 2024-03-25T18:14:32.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:32.523+0000 7f1a7b38c640 1 librados: shutdown 2024-03-25T18:14:32.537 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:33.256 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:14.948 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:33.264 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:14.957 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:33.302 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:14.994 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:33.315 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:02:14.975 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:33.328 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:02:14.989 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:33.329 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:15.021 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:33.340 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:15.032 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:33.347 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:02:15.008 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:33.368 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:15.061 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:39.344 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:39.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.439+0000 7f2f8ae31640 10 monclient: get_monmap_and_config 2024-03-25T18:14:39.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.439+0000 7f2f8ae31640 10 monclient: build_initial_monmap 2024-03-25T18:14:39.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.439+0000 7f2f8ae31640 10 monclient: monmap: 2024-03-25T18:14:39.441 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:14:39.441 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:39.441 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:14:39.440898+0000 2024-03-25T18:14:39.441 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:14:39.440898+0000 2024-03-25T18:14:39.441 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:39.441 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:39.441 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:14:39.441 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:14:39.441 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:14:39.441 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:39.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.440+0000 7f2f8ae31640 1 Processor -- start 2024-03-25T18:14:39.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.440+0000 7f2f8ae31640 1 -- start start 2024-03-25T18:14:39.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.440+0000 7f2f8ae31640 10 monclient: init 2024-03-25T18:14:39.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.441+0000 7f2f8ae31640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:39.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.441+0000 7f2f8ae31640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:39.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.441+0000 7f2f8ae31640 10 monclient(hunting): picked mon.noname-c con 0x7f2f840b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T18:14:39.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.441+0000 7f2f8ae31640 10 monclient(hunting): picked mon.noname-a con 0x7f2f840b1730 addr v1:172.21.15.63:6789/0 2024-03-25T18:14:39.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.441+0000 7f2f8ae31640 10 monclient(hunting): picked mon.noname-b con 0x7f2f8414be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:14:39.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.441+0000 7f2f8ae31640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f840af940 con 0x7f2f840b1730 2024-03-25T18:14:39.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.441+0000 7f2f8ae31640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f840b0a70 con 0x7f2f840b4d60 2024-03-25T18:14:39.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.441+0000 7f2f8ae31640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f840b0d50 con 0x7f2f8414be90 2024-03-25T18:14:39.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.441+0000 7f2f8ae31640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:39.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.441+0000 7f2f8ae31640 10 monclient(hunting): authenticate will time out at 7573.627569s 2024-03-25T18:14:39.455 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.454+0000 7f2f83fff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f2f840b1730 0x7f2f840b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:38680/0 (socket says 172.21.15.63:38680) 2024-03-25T18:14:39.455 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.454+0000 7f2f83fff640 1 -- 172.21.15.63:0/49096749 learned_addr learned my addr 172.21.15.63:0/49096749 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:14:39.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.476+0000 7f2f837fe640 1 -- 172.21.15.63:0/49096749 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657045511 0 0) 0x7f2f840af940 con 0x7f2f840b1730 2024-03-25T18:14:39.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.476+0000 7f2f837fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:39.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.476+0000 7f2f837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:39.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.476+0000 7f2f837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:39.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.476+0000 7f2f837fe640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:14:39.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.476+0000 7f2f837fe640 1 -- 172.21.15.63:0/49096749 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f74003520 con 0x7f2f840b1730 2024-03-25T18:14:39.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.488+0000 7f2f837fe640 1 -- 172.21.15.63:0/49096749 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569164658 0 0) 0x7f2f840b0a70 con 0x7f2f840b4d60 2024-03-25T18:14:39.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.488+0000 7f2f837fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:39.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.488+0000 7f2f837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:39.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.488+0000 7f2f837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:39.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.488+0000 7f2f837fe640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:14:39.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.488+0000 7f2f837fe640 1 -- 172.21.15.63:0/49096749 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f840af940 con 0x7f2f840b4d60 2024-03-25T18:14:39.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.489+0000 7f2f837fe640 1 -- 172.21.15.63:0/49096749 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533340293 0 0) 0x7f2f840b0d50 con 0x7f2f8414be90 2024-03-25T18:14:39.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.489+0000 7f2f837fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:39.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.489+0000 7f2f837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:39.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.489+0000 7f2f837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:39.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.489+0000 7f2f837fe640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:14:39.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.489+0000 7f2f837fe640 1 -- 172.21.15.63:0/49096749 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f840b0a70 con 0x7f2f8414be90 2024-03-25T18:14:39.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.494+0000 7f2f837fe640 1 -- 172.21.15.63:0/49096749 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 373983374 0 0) 0x7f2f840b0a70 con 0x7f2f8414be90 2024-03-25T18:14:39.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.494+0000 7f2f837fe640 1 -- 172.21.15.63:0/49096749 >> v1:172.21.15.81:6789/0 conn(0x7f2f840b4d60 legacy=0x7f2f8413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:39.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.494+0000 7f2f837fe640 1 -- 172.21.15.63:0/49096749 >> v1:172.21.15.63:6789/0 conn(0x7f2f840b1730 legacy=0x7f2f840b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:39.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.494+0000 7f2f837fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:39.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.494+0000 7f2f837fe640 1 monclient: found mon.noname-b 2024-03-25T18:14:39.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.494+0000 7f2f837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:39.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.494+0000 7f2f837fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:14:39.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.494+0000 7f2f837fe640 1 -- 172.21.15.63:0/49096749 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f840b1030 con 0x7f2f8414be90 2024-03-25T18:14:39.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.494+0000 7f2f837fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:39.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.494+0000 7f2f837fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:39.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.494+0000 7f2f837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:39.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.494+0000 7f2f837fe640 1 -- 172.21.15.63:0/49096749 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2f7c001930 con 0x7f2f8414be90 2024-03-25T18:14:39.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.494+0000 7f2f837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:39.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.495+0000 7f2f837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:39.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.495+0000 7f2f837fe640 10 monclient: dump: 2024-03-25T18:14:39.497 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:39.497 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:39.497 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:39.497 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:39.497 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:39.497 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:39.498 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:39.498 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:39.498 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:39.498 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:39.498 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.495+0000 7f2f8ae31640 5 monclient: authenticate success, global_id 4313 2024-03-25T18:14:39.498 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.495+0000 7f2f8ae31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:39.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.510+0000 7f2f837fe640 1 -- 172.21.15.63:0/49096749 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f7c001a90 con 0x7f2f8414be90 2024-03-25T18:14:39.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.510+0000 7f2f837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:39.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.510+0000 7f2f837fe640 1 -- 172.21.15.63:0/49096749 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2f7c001ee0 con 0x7f2f8414be90 2024-03-25T18:14:39.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.510+0000 7f2f8ae31640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:39.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.510+0000 7f2f8ae31640 4 set_mon_vals no callback set 2024-03-25T18:14:39.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.510+0000 7f2f837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:39.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.510+0000 7f2f837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:39.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.510+0000 7f2f837fe640 10 monclient: dump: 2024-03-25T18:14:39.512 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:39.512 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:39.512 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:39.512 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:39.512 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:39.512 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:39.512 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:39.512 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:39.512 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:39.512 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:39.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.511+0000 7f2f8ae31640 10 monclient: shutdown 2024-03-25T18:14:39.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.511+0000 7f2f8ae31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:39.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.511+0000 7f2f8ae31640 1 -- 172.21.15.63:0/49096749 >> v1:172.21.15.63:6790/0 conn(0x7f2f8414be90 legacy=0x7f2f8414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:39.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.511+0000 7f2f8ae31640 1 -- 172.21.15.63:0/49096749 shutdown_connections 2024-03-25T18:14:39.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.511+0000 7f2f8ae31640 1 -- 172.21.15.63:0/49096749 >> 172.21.15.63:0/49096749 conn(0x7f2f840617a0 msgr2=0x7f2f84061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:39.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.511+0000 7f2f8ae31640 1 -- 172.21.15.63:0/49096749 shutdown_connections 2024-03-25T18:14:39.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.511+0000 7f2f8ae31640 1 -- 172.21.15.63:0/49096749 wait complete. 2024-03-25T18:14:39.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.512+0000 7f2f8ae31640 10 monclient: build_initial_monmap 2024-03-25T18:14:39.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.512+0000 7f2f8ae31640 10 monclient: monmap: 2024-03-25T18:14:39.514 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:14:39.514 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:39.515 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:14:39.515 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:14:39.515 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:39.515 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:14:39.515 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:14:39.515 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:14:39.515 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:14:39.515 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:39.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.512+0000 7f2f8ae31640 1 librados: starting msgr at 2024-03-25T18:14:39.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.512+0000 7f2f8ae31640 1 librados: starting objecter 2024-03-25T18:14:39.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.512+0000 7f2f8ae31640 1 Processor -- start 2024-03-25T18:14:39.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.512+0000 7f2f8ae31640 1 -- start start 2024-03-25T18:14:39.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.512+0000 7f2f8ae31640 1 librados: setting wanted keys 2024-03-25T18:14:39.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.512+0000 7f2f8ae31640 1 librados: calling monclient init 2024-03-25T18:14:39.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.512+0000 7f2f8ae31640 10 monclient: init 2024-03-25T18:14:39.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.513+0000 7f2f8ae31640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:39.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.513+0000 7f2f8ae31640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:39.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.514+0000 7f2f8ae31640 10 monclient(hunting): picked mon.noname-c con 0x7f2f840b1730 addr v1:172.21.15.63:6790/0 2024-03-25T18:14:39.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.514+0000 7f2f8ae31640 10 monclient(hunting): picked mon.noname-b con 0x7f2f840b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T18:14:39.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.514+0000 7f2f8ae31640 10 monclient(hunting): picked mon.noname-a con 0x7f2f8414be90 addr v1:172.21.15.63:6789/0 2024-03-25T18:14:39.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.514+0000 7f2f8ae31640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f84140ce0 con 0x7f2f8414be90 2024-03-25T18:14:39.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.514+0000 7f2f8ae31640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f841382d0 con 0x7f2f840b4d60 2024-03-25T18:14:39.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.514+0000 7f2f8ae31640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f84138460 con 0x7f2f840b1730 2024-03-25T18:14:39.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.514+0000 7f2f8ae31640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:39.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.514+0000 7f2f8ae31640 10 monclient(hunting): authenticate will time out at 7573.700474s 2024-03-25T18:14:39.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.520+0000 7f2f893ce640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f2f8414be90 0x7f2f84142630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:38728/0 (socket says 172.21.15.63:38728) 2024-03-25T18:14:39.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.520+0000 7f2f893ce640 1 -- 172.21.15.63:0/576382828 learned_addr learned my addr 172.21.15.63:0/576382828 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:14:39.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.526+0000 7f2f817fa640 10 client.?.objecter ms_handle_connect 0x7f2f840b4d60 2024-03-25T18:14:39.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.526+0000 7f2f817fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:39.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.528+0000 7f2f817fa640 10 client.?.objecter ms_handle_connect 0x7f2f8414be90 2024-03-25T18:14:39.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.528+0000 7f2f817fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:39.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.534+0000 7f2f817fa640 10 client.?.objecter ms_handle_connect 0x7f2f840b1730 2024-03-25T18:14:39.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.534+0000 7f2f817fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:39.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.535+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020294857 0 0) 0x7f2f841382d0 con 0x7f2f840b4d60 2024-03-25T18:14:39.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.535+0000 7f2f817fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:39.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.535+0000 7f2f817fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:39.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.535+0000 7f2f817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:39.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.535+0000 7f2f817fa640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:14:39.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.535+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f68004eb0 con 0x7f2f840b4d60 2024-03-25T18:14:39.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.543+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482569704 0 0) 0x7f2f84140ce0 con 0x7f2f8414be90 2024-03-25T18:14:39.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.543+0000 7f2f817fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:39.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.543+0000 7f2f817fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:39.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.543+0000 7f2f817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:39.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.543+0000 7f2f817fa640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:14:39.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.543+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f841382d0 con 0x7f2f8414be90 2024-03-25T18:14:39.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.545+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187198731 0 0) 0x7f2f840af940 con 0x7f2f840b1730 2024-03-25T18:14:39.546 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.545+0000 7f2f817fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:39.546 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.545+0000 7f2f817fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:39.546 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.545+0000 7f2f817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:39.546 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.545+0000 7f2f817fa640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:14:39.546 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.545+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f84140ce0 con 0x7f2f840b1730 2024-03-25T18:14:39.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.557+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 372178386 0 0) 0x7f2f68004eb0 con 0x7f2f840b4d60 2024-03-25T18:14:39.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.557+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f840af940 con 0x7f2f840b4d60 2024-03-25T18:14:39.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.557+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2f70001950 con 0x7f2f840b4d60 2024-03-25T18:14:39.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.557+0000 7f2f817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:39.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.557+0000 7f2f817fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:39.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.557+0000 7f2f817fa640 10 monclient(hunting): dump: 2024-03-25T18:14:39.558 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:39.558 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:39.559 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:39.559 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:39.559 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:39.559 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:39.559 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:39.559 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:39.559 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:39.559 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:39.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.557+0000 7f2f817fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:39.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.561+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 465607402 0 0) 0x7f2f84140ce0 con 0x7f2f840b1730 2024-03-25T18:14:39.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.561+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f68004eb0 con 0x7f2f840b1730 2024-03-25T18:14:39.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.561+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2f78001790 con 0x7f2f840b1730 2024-03-25T18:14:39.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.561+0000 7f2f817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:39.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.561+0000 7f2f817fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:39.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.561+0000 7f2f817fa640 10 monclient(hunting): dump: 2024-03-25T18:14:39.562 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:39.562 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:39.562 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:39.562 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:39.562 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:39.563 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:39.563 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:39.563 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:39.563 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:39.563 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:39.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2533183941 0 0) 0x7f2f840af940 con 0x7f2f840b4d60 2024-03-25T18:14:39.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 >> v1:172.21.15.63:6790/0 conn(0x7f2f840b1730 legacy=0x7f2f84140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:39.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 >> v1:172.21.15.63:6789/0 conn(0x7f2f8414be90 legacy=0x7f2f84142630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:39.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f817fa640 10 monclient: _finish_hunting 0 2024-03-25T18:14:39.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f817fa640 1 monclient: found mon.b 2024-03-25T18:14:39.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:39.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f817fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:39.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f841324b0 con 0x7f2f840b4d60 2024-03-25T18:14:39.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f817fa640 10 monclient: _finish_auth 0 2024-03-25T18:14:39.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f817fa640 10 monclient: _check_auth_tickets 2024-03-25T18:14:39.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:39.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f8ae31640 5 monclient: authenticate success, global_id 4318 2024-03-25T18:14:39.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f8ae31640 10 monclient: _renew_subs 2024-03-25T18:14:39.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f8ae31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:39.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f8ae31640 1 -- 172.21.15.63:0/576382828 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f84132760 con 0x7f2f840b4d60 2024-03-25T18:14:39.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f8ae31640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:39.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f8ae31640 10 monclient: _renew_subs 2024-03-25T18:14:39.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f8ae31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:39.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f8ae31640 1 -- 172.21.15.63:0/576382828 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f84144d80 con 0x7f2f840b4d60 2024-03-25T18:14:39.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.577+0000 7f2f8ae31640 1 librados: init done 2024-03-25T18:14:39.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.578+0000 7f2f8ae31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:39.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.578+0000 7f2f8ae31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:39.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.578+0000 7f2f8ae31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:39.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.578+0000 7f2f8ae31640 1 -- 172.21.15.63:0/576382828 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f84140b20 con 0x7f2f840b4d60 2024-03-25T18:14:39.582 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.582+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f70001d60 con 0x7f2f840b4d60 2024-03-25T18:14:39.582 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.582+0000 7f2f817fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:39.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.582+0000 7f2f82ffd640 4 set_mon_vals no callback set 2024-03-25T18:14:39.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.584+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2f70004980 con 0x7f2f840b4d60 2024-03-25T18:14:39.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.584+0000 7f2f817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:39.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.584+0000 7f2f817fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:39.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.584+0000 7f2f817fa640 10 monclient: dump: 2024-03-25T18:14:39.586 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:39.586 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:39.586 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:39.586 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:39.586 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:39.586 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:39.586 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:39.586 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:39.586 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:39.586 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:39.870 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.869+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f2f7001a9d0 con 0x7f2f840b4d60 2024-03-25T18:14:39.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.870+0000 7f2f88bcd640 10 monclient: get_auth_request con 0x7f2f6806df70 auth_method 0 2024-03-25T18:14:39.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.870+0000 7f2f817fa640 10 client.4318.objecter ms_handle_connect 0x7f2f6806df70 2024-03-25T18:14:39.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.875+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f2f700547a0 con 0x7f2f840b4d60 2024-03-25T18:14:39.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.876+0000 7f2f817fa640 10 client.4318.objecter ms_dispatch 0x7f2f84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:39.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.876+0000 7f2f817fa640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:39.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.876+0000 7f2f817fa640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:39.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.876+0000 7f2f817fa640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-25T18:14:39.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.876+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f700500e0 con 0x7f2f840b4d60 2024-03-25T18:14:39.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.876+0000 7f2f817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:39.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:39.876+0000 7f2f817fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:40.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.128+0000 7f2f8ae31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:40.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.128+0000 7f2f8ae31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:40.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.128+0000 7f2f8ae31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:40.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.128+0000 7f2f8ae31640 1 -- 172.21.15.63:0/576382828 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f840627b0 con 0x7f2f840b4d60 2024-03-25T18:14:40.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.157+0000 7f2f817fa640 1 -- 172.21.15.63:0/576382828 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f2f700521e0 con 0x7f2f840b4d60 2024-03-25T18:14:40.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.157+0000 7f2f817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:40.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.157+0000 7f2f817fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:40.161 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:14:40.161 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:40.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.160+0000 7f2f8ae31640 10 librados: watch_flush enter 2024-03-25T18:14:40.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.160+0000 7f2f8ae31640 10 librados: watch_flush exit 2024-03-25T18:14:40.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.160+0000 7f2f8ae31640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-25T18:14:40.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.160+0000 7f2f8ae31640 10 client.4318.objecter successfully canceled tick 2024-03-25T18:14:40.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.160+0000 7f2f8ae31640 1 -- 172.21.15.63:0/576382828 >> v1:172.21.15.81:6800/3640340873 conn(0x7f2f6806df70 legacy=0x7f2f68070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.160+0000 7f2f8ae31640 10 monclient: shutdown 2024-03-25T18:14:40.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.160+0000 7f2f8ae31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:40.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.160+0000 7f2f8ae31640 1 -- 172.21.15.63:0/576382828 >> v1:172.21.15.81:6789/0 conn(0x7f2f840b4d60 legacy=0x7f2f84131da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.160+0000 7f2f8ae31640 1 -- 172.21.15.63:0/576382828 shutdown_connections 2024-03-25T18:14:40.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.160+0000 7f2f8ae31640 1 -- 172.21.15.63:0/576382828 >> 172.21.15.63:0/576382828 conn(0x7f2f840617a0 msgr2=0x7f2f840b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:40.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.160+0000 7f2f8ae31640 1 -- 172.21.15.63:0/576382828 shutdown_connections 2024-03-25T18:14:40.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.160+0000 7f2f8ae31640 1 -- 172.21.15.63:0/576382828 wait complete. 2024-03-25T18:14:40.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:40.160+0000 7f2f8ae31640 1 librados: shutdown 2024-03-25T18:14:40.174 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:46.981 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:47.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.097+0000 7fcf05a0e640 10 monclient: get_monmap_and_config 2024-03-25T18:14:47.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.097+0000 7fcf05a0e640 10 monclient: build_initial_monmap 2024-03-25T18:14:47.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.097+0000 7fcf05a0e640 10 monclient: monmap: 2024-03-25T18:14:47.098 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:14:47.098 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:47.098 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:14:47.098460+0000 2024-03-25T18:14:47.098 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:14:47.098460+0000 2024-03-25T18:14:47.098 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:47.099 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:47.099 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:14:47.099 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:14:47.099 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:14:47.099 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:47.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.098+0000 7fcf05a0e640 1 Processor -- start 2024-03-25T18:14:47.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.098+0000 7fcf05a0e640 1 -- start start 2024-03-25T18:14:47.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.098+0000 7fcf05a0e640 10 monclient: init 2024-03-25T18:14:47.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.098+0000 7fcf05a0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:47.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.098+0000 7fcf05a0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:47.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.098+0000 7fcf05a0e640 10 monclient(hunting): picked mon.noname-c con 0x7fcf000a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:14:47.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.098+0000 7fcf05a0e640 10 monclient(hunting): picked mon.noname-a con 0x7fcf0013f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:14:47.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.098+0000 7fcf05a0e640 10 monclient(hunting): picked mon.noname-b con 0x7fcf001430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:14:47.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.098+0000 7fcf05a0e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf00142520 con 0x7fcf0013f710 2024-03-25T18:14:47.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.098+0000 7fcf05a0e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf000b28a0 con 0x7fcf000a98f0 2024-03-25T18:14:47.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.098+0000 7fcf05a0e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf000b2a30 con 0x7fcf001430c0 2024-03-25T18:14:47.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.098+0000 7fcf05a0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:47.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.098+0000 7fcf05a0e640 10 monclient(hunting): authenticate will time out at 7581.285132s 2024-03-25T18:14:47.106 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.105+0000 7fcefeffd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fcf000a98f0 0x7fcf000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:35806/0 (socket says 172.21.15.63:35806) 2024-03-25T18:14:47.106 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.105+0000 7fcefeffd640 1 -- 172.21.15.63:0/2413567716 learned_addr learned my addr 172.21.15.63:0/2413567716 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:14:47.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.115+0000 7fcefdffb640 1 -- 172.21.15.63:0/2413567716 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606750767 0 0) 0x7fcf000b28a0 con 0x7fcf000a98f0 2024-03-25T18:14:47.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.115+0000 7fcefdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:47.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.115+0000 7fcefdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:47.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.115+0000 7fcefdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:47.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.115+0000 7fcefdffb640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:14:47.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.115+0000 7fcefdffb640 1 -- 172.21.15.63:0/2413567716 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee8003520 con 0x7fcf000a98f0 2024-03-25T18:14:47.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.124+0000 7fcefdffb640 1 -- 172.21.15.63:0/2413567716 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168603488 0 0) 0x7fcf000b2a30 con 0x7fcf001430c0 2024-03-25T18:14:47.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.124+0000 7fcefdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:47.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.124+0000 7fcefdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:47.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.124+0000 7fcefdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:47.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.124+0000 7fcefdffb640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:14:47.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.124+0000 7fcefdffb640 1 -- 172.21.15.63:0/2413567716 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf000b28a0 con 0x7fcf001430c0 2024-03-25T18:14:47.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.124+0000 7fcefdffb640 1 -- 172.21.15.63:0/2413567716 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661093883 0 0) 0x7fcf00142520 con 0x7fcf0013f710 2024-03-25T18:14:47.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.124+0000 7fcefdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:47.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.124+0000 7fcefdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:47.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.124+0000 7fcefdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:47.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.124+0000 7fcefdffb640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:14:47.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.124+0000 7fcefdffb640 1 -- 172.21.15.63:0/2413567716 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf000b2a30 con 0x7fcf0013f710 2024-03-25T18:14:47.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcefdffb640 1 -- 172.21.15.63:0/2413567716 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3424037422 0 0) 0x7fcee8003520 con 0x7fcf000a98f0 2024-03-25T18:14:47.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcefdffb640 1 -- 172.21.15.63:0/2413567716 >> v1:172.21.15.63:6790/0 conn(0x7fcf001430c0 legacy=0x7fcf0014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:47.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcefdffb640 1 -- 172.21.15.63:0/2413567716 >> v1:172.21.15.63:6789/0 conn(0x7fcf0013f710 legacy=0x7fcf001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:47.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcefdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:47.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcefdffb640 1 monclient: found mon.noname-c 2024-03-25T18:14:47.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcefdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:47.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcefdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:14:47.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcefdffb640 1 -- 172.21.15.63:0/2413567716 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf000b44c0 con 0x7fcf000a98f0 2024-03-25T18:14:47.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcefdffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:47.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcefdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:47.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcefdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:47.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcefdffb640 1 -- 172.21.15.63:0/2413567716 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fcef0001690 con 0x7fcf000a98f0 2024-03-25T18:14:47.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcefdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:47.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcefdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:47.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcefdffb640 10 monclient: dump: 2024-03-25T18:14:47.131 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:47.131 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:47.131 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:47.132 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:47.132 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:47.132 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:47.132 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:47.132 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:47.132 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:47.132 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:47.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcf05a0e640 5 monclient: authenticate success, global_id 4321 2024-03-25T18:14:47.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.129+0000 7fcf05a0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:47.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.142+0000 7fcefdffb640 1 -- 172.21.15.63:0/2413567716 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcef00017f0 con 0x7fcf000a98f0 2024-03-25T18:14:47.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.142+0000 7fcefdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:47.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.142+0000 7fcefdffb640 1 -- 172.21.15.63:0/2413567716 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fcef0001c00 con 0x7fcf000a98f0 2024-03-25T18:14:47.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.142+0000 7fcefdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:47.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.142+0000 7fcefdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:47.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.142+0000 7fcefdffb640 10 monclient: dump: 2024-03-25T18:14:47.143 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:47.143 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:47.144 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:47.144 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:47.144 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:47.144 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:47.144 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:47.144 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:47.144 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:47.144 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:47.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.142+0000 7fcf05a0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:47.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.142+0000 7fcf05a0e640 4 set_mon_vals no callback set 2024-03-25T18:14:47.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.143+0000 7fcf05a0e640 10 monclient: shutdown 2024-03-25T18:14:47.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.143+0000 7fcf05a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:47.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.143+0000 7fcf05a0e640 1 -- 172.21.15.63:0/2413567716 >> v1:172.21.15.81:6789/0 conn(0x7fcf000a98f0 legacy=0x7fcf000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:47.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.143+0000 7fcf05a0e640 1 -- 172.21.15.63:0/2413567716 shutdown_connections 2024-03-25T18:14:47.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.143+0000 7fcf05a0e640 1 -- 172.21.15.63:0/2413567716 >> 172.21.15.63:0/2413567716 conn(0x7fcf000b7ea0 msgr2=0x7fcf000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:47.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.143+0000 7fcf05a0e640 1 -- 172.21.15.63:0/2413567716 shutdown_connections 2024-03-25T18:14:47.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.143+0000 7fcf05a0e640 1 -- 172.21.15.63:0/2413567716 wait complete. 2024-03-25T18:14:47.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.144+0000 7fcf05a0e640 10 monclient: build_initial_monmap 2024-03-25T18:14:47.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.144+0000 7fcf05a0e640 10 monclient: monmap: 2024-03-25T18:14:47.146 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:14:47.146 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:47.146 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:14:47.146 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:14:47.146 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:47.147 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:14:47.147 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:14:47.147 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:14:47.147 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:14:47.147 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:47.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.144+0000 7fcf05a0e640 1 librados: starting msgr at 2024-03-25T18:14:47.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.144+0000 7fcf05a0e640 1 librados: starting objecter 2024-03-25T18:14:47.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.144+0000 7fcf05a0e640 1 Processor -- start 2024-03-25T18:14:47.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.145+0000 7fcf05a0e640 1 -- start start 2024-03-25T18:14:47.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.145+0000 7fcf05a0e640 1 librados: setting wanted keys 2024-03-25T18:14:47.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.145+0000 7fcf05a0e640 1 librados: calling monclient init 2024-03-25T18:14:47.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.145+0000 7fcf05a0e640 10 monclient: init 2024-03-25T18:14:47.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.145+0000 7fcf05a0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:47.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.145+0000 7fcf05a0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:47.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.145+0000 7fcf05a0e640 10 monclient(hunting): picked mon.noname-b con 0x7fcf000a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:14:47.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.145+0000 7fcf05a0e640 10 monclient(hunting): picked mon.noname-a con 0x7fcf0013f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:14:47.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.145+0000 7fcf05a0e640 10 monclient(hunting): picked mon.noname-c con 0x7fcf001430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:14:47.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.145+0000 7fcf05a0e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf00137ab0 con 0x7fcf0013f710 2024-03-25T18:14:47.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.146+0000 7fcf05a0e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf00138bd0 con 0x7fcf000a98f0 2024-03-25T18:14:47.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.146+0000 7fcf05a0e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf00138e40 con 0x7fcf001430c0 2024-03-25T18:14:47.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.146+0000 7fcf05a0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:47.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.146+0000 7fcf05a0e640 10 monclient(hunting): authenticate will time out at 7581.332381s 2024-03-25T18:14:47.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.157+0000 7fcefe7fc640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fcf0013f710 0x7fcf001df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:43304/0 (socket says 172.21.15.63:43304) 2024-03-25T18:14:47.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.157+0000 7fcefe7fc640 1 -- 172.21.15.63:0/855714686 learned_addr learned my addr 172.21.15.63:0/855714686 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:14:47.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.158+0000 7fcee7fff640 10 client.?.objecter ms_handle_connect 0x7fcf0013f710 2024-03-25T18:14:47.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.158+0000 7fcee7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:47.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.162+0000 7fcee7fff640 10 client.?.objecter ms_handle_connect 0x7fcf001430c0 2024-03-25T18:14:47.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.162+0000 7fcee7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:47.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.167+0000 7fcee7fff640 10 client.?.objecter ms_handle_connect 0x7fcf000a98f0 2024-03-25T18:14:47.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.167+0000 7fcee7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:47.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.173+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343502203 0 0) 0x7fcf00137ab0 con 0x7fcf0013f710 2024-03-25T18:14:47.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.174+0000 7fcee7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:47.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.174+0000 7fcee7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:47.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.174+0000 7fcee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:47.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.174+0000 7fcee7fff640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:14:47.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.174+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceec004eb0 con 0x7fcf0013f710 2024-03-25T18:14:47.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.174+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249025652 0 0) 0x7fcf00138e40 con 0x7fcf001430c0 2024-03-25T18:14:47.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.174+0000 7fcee7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:47.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.174+0000 7fcee7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:47.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.174+0000 7fcee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:47.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.174+0000 7fcee7fff640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:14:47.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.174+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf00137ab0 con 0x7fcf001430c0 2024-03-25T18:14:47.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.175+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856742391 0 0) 0x7fcf00138bd0 con 0x7fcf000a98f0 2024-03-25T18:14:47.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.175+0000 7fcee7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:47.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.175+0000 7fcee7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:47.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.175+0000 7fcee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:47.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.175+0000 7fcee7fff640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:14:47.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.175+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf00138e40 con 0x7fcf000a98f0 2024-03-25T18:14:47.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.187+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2900912156 0 0) 0x7fceec004eb0 con 0x7fcf0013f710 2024-03-25T18:14:47.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.187+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf00138bd0 con 0x7fcf0013f710 2024-03-25T18:14:47.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.187+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fcef4001a10 con 0x7fcf0013f710 2024-03-25T18:14:47.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.187+0000 7fcee7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:47.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.187+0000 7fcee7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:47.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.187+0000 7fcee7fff640 10 monclient(hunting): dump: 2024-03-25T18:14:47.189 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:47.189 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:47.189 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:47.189 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:47.189 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:47.189 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:47.189 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:47.190 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:47.190 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:47.190 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:47.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.187+0000 7fcee7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:47.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.188+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3879935539 0 0) 0x7fcf00138e40 con 0x7fcf000a98f0 2024-03-25T18:14:47.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.188+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fceec004eb0 con 0x7fcf000a98f0 2024-03-25T18:14:47.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.188+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fcef0001b00 con 0x7fcf000a98f0 2024-03-25T18:14:47.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.188+0000 7fcee7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:47.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.188+0000 7fcee7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:47.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.188+0000 7fcee7fff640 10 monclient(hunting): dump: 2024-03-25T18:14:47.190 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:47.190 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:47.190 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:47.190 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:47.191 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:47.191 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:47.191 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:47.191 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:47.191 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:47.191 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:47.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1443738980 0 0) 0x7fceec004eb0 con 0x7fcf000a98f0 2024-03-25T18:14:47.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 >> v1:172.21.15.63:6790/0 conn(0x7fcf001430c0 legacy=0x7fcf001e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:47.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 >> v1:172.21.15.63:6789/0 conn(0x7fcf0013f710 legacy=0x7fcf001df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:47.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcee7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:47.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcee7fff640 1 monclient: found mon.b 2024-03-25T18:14:47.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcee7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:47.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcee7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:47.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf00139120 con 0x7fcf000a98f0 2024-03-25T18:14:47.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcee7fff640 10 monclient: _finish_auth 0 2024-03-25T18:14:47.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcee7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:47.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcee7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:47.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcf05a0e640 5 monclient: authenticate success, global_id 4324 2024-03-25T18:14:47.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcf05a0e640 10 monclient: _renew_subs 2024-03-25T18:14:47.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcf05a0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:47.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcf05a0e640 1 -- 172.21.15.63:0/855714686 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf00139400 con 0x7fcf000a98f0 2024-03-25T18:14:47.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcf05a0e640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:47.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcf05a0e640 10 monclient: _renew_subs 2024-03-25T18:14:47.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcf05a0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:47.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcf05a0e640 1 -- 172.21.15.63:0/855714686 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf000b7880 con 0x7fcf000a98f0 2024-03-25T18:14:47.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.194+0000 7fcf05a0e640 1 librados: init done 2024-03-25T18:14:47.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.195+0000 7fcf05a0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:47.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.195+0000 7fcf05a0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:47.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.195+0000 7fcf05a0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:47.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.195+0000 7fcf05a0e640 1 -- 172.21.15.63:0/855714686 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf00137810 con 0x7fcf000a98f0 2024-03-25T18:14:47.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.464+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcef0001c60 con 0x7fcf000a98f0 2024-03-25T18:14:47.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.464+0000 7fcee7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:47.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.464+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fcef0004c30 con 0x7fcf000a98f0 2024-03-25T18:14:47.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.464+0000 7fcee7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:47.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.464+0000 7fcefd7fa640 4 set_mon_vals no callback set 2024-03-25T18:14:47.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.464+0000 7fcee7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:47.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.464+0000 7fcee7fff640 10 monclient: dump: 2024-03-25T18:14:47.466 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:47.466 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:47.466 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:47.466 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:47.466 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:47.466 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:47.466 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:47.466 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:47.466 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:47.466 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:47.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.465+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fcef001ac00 con 0x7fcf000a98f0 2024-03-25T18:14:47.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.465+0000 7fcefe7fc640 10 monclient: get_auth_request con 0x7fceec06df70 auth_method 0 2024-03-25T18:14:47.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.466+0000 7fcee7fff640 10 client.4324.objecter ms_handle_connect 0x7fceec06df70 2024-03-25T18:14:47.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.473+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fcef0054800 con 0x7fcf000a98f0 2024-03-25T18:14:47.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.473+0000 7fcee7fff640 10 client.4324.objecter ms_dispatch 0x7fcf00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:47.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.473+0000 7fcee7fff640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:47.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.473+0000 7fcee7fff640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:47.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.473+0000 7fcee7fff640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-25T18:14:47.475 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.474+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcef0050310 con 0x7fcf000a98f0 2024-03-25T18:14:47.475 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.474+0000 7fcee7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:47.475 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.474+0000 7fcee7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:47.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.702+0000 7fcf05a0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:47.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.702+0000 7fcf05a0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:47.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.702+0000 7fcf05a0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:47.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.702+0000 7fcf05a0e640 1 -- 172.21.15.63:0/855714686 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf000627b0 con 0x7fcf000a98f0 2024-03-25T18:14:47.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.723+0000 7fcee7fff640 1 -- 172.21.15.63:0/855714686 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fcef0052240 con 0x7fcf000a98f0 2024-03-25T18:14:47.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.724+0000 7fcee7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:47.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.724+0000 7fcee7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:47.725 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:14:47.725 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:47.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.726+0000 7fcf05a0e640 10 librados: watch_flush enter 2024-03-25T18:14:47.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.726+0000 7fcf05a0e640 10 librados: watch_flush exit 2024-03-25T18:14:47.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.726+0000 7fcf05a0e640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-25T18:14:47.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.726+0000 7fcf05a0e640 10 client.4324.objecter successfully canceled tick 2024-03-25T18:14:47.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.726+0000 7fcf05a0e640 1 -- 172.21.15.63:0/855714686 >> v1:172.21.15.81:6800/3640340873 conn(0x7fceec06df70 legacy=0x7fceec070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:47.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.726+0000 7fcf05a0e640 10 monclient: shutdown 2024-03-25T18:14:47.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.726+0000 7fcf05a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:47.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.726+0000 7fcf05a0e640 1 -- 172.21.15.63:0/855714686 >> v1:172.21.15.81:6789/0 conn(0x7fcf000a98f0 legacy=0x7fcf00137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:47.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.726+0000 7fcf05a0e640 1 -- 172.21.15.63:0/855714686 shutdown_connections 2024-03-25T18:14:47.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.726+0000 7fcf05a0e640 1 -- 172.21.15.63:0/855714686 >> 172.21.15.63:0/855714686 conn(0x7fcf000b7ea0 msgr2=0x7fcf00074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:47.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.727+0000 7fcf05a0e640 1 -- 172.21.15.63:0/855714686 shutdown_connections 2024-03-25T18:14:47.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.727+0000 7fcf05a0e640 1 -- 172.21.15.63:0/855714686 wait complete. 2024-03-25T18:14:47.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:47.727+0000 7fcf05a0e640 1 librados: shutdown 2024-03-25T18:14:47.740 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:53.304 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:34.997 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:53.367 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:35.059 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:53.399 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:35.091 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:53.421 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:35.114 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:53.432 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:35.124 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:53.458 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:35.151 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:53.459 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:02:35.120 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:53.463 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:02:35.124 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:53.484 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:02:35.145 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:54.547 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:54.647 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.645+0000 7ffabc904640 10 monclient: get_monmap_and_config 2024-03-25T18:14:54.647 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.645+0000 7ffabc904640 10 monclient: build_initial_monmap 2024-03-25T18:14:54.647 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.645+0000 7ffabc904640 10 monclient: monmap: 2024-03-25T18:14:54.647 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:14:54.647 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:54.647 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:14:54.646978+0000 2024-03-25T18:14:54.647 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:14:54.646978+0000 2024-03-25T18:14:54.647 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:54.647 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:54.647 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:14:54.647 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:14:54.647 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:14:54.647 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:54.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.646+0000 7ffabc904640 1 Processor -- start 2024-03-25T18:14:54.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.646+0000 7ffabc904640 1 -- start start 2024-03-25T18:14:54.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.646+0000 7ffabc904640 10 monclient: init 2024-03-25T18:14:54.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.647+0000 7ffabc904640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:54.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.647+0000 7ffabc904640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:54.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.647+0000 7ffabc904640 10 monclient(hunting): picked mon.noname-b con 0x7ffab80a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:14:54.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.647+0000 7ffabc904640 10 monclient(hunting): picked mon.noname-c con 0x7ffab813f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:14:54.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.647+0000 7ffabc904640 10 monclient(hunting): picked mon.noname-a con 0x7ffab81430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:14:54.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.647+0000 7ffabc904640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab8142520 con 0x7ffab81430c0 2024-03-25T18:14:54.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.647+0000 7ffabc904640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab80b28a0 con 0x7ffab813f710 2024-03-25T18:14:54.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.647+0000 7ffabc904640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab80b2a30 con 0x7ffab80a98f0 2024-03-25T18:14:54.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.647+0000 7ffabc904640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:54.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.647+0000 7ffabc904640 10 monclient(hunting): authenticate will time out at 7588.834022s 2024-03-25T18:14:54.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.649+0000 7ffab5d9b640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ffab813f710 0x7ffab81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:33876/0 (socket says 172.21.15.63:33876) 2024-03-25T18:14:54.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.649+0000 7ffab5d9b640 1 -- 172.21.15.63:0/1877572979 learned_addr learned my addr 172.21.15.63:0/1877572979 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:14:54.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.654+0000 7ffab559a640 1 -- 172.21.15.63:0/1877572979 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927712847 0 0) 0x7ffab80b28a0 con 0x7ffab813f710 2024-03-25T18:14:54.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.654+0000 7ffab559a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:54.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.654+0000 7ffab559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:54.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.654+0000 7ffab559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:54.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.654+0000 7ffab559a640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:14:54.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.654+0000 7ffab559a640 1 -- 172.21.15.63:0/1877572979 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa0003520 con 0x7ffab813f710 2024-03-25T18:14:54.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.660+0000 7ffab559a640 1 -- 172.21.15.63:0/1877572979 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3109196669 0 0) 0x7ffaa0003520 con 0x7ffab813f710 2024-03-25T18:14:54.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.660+0000 7ffab559a640 1 -- 172.21.15.63:0/1877572979 >> v1:172.21.15.63:6790/0 conn(0x7ffab80a98f0 legacy=0x7ffab80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:14:54.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.660+0000 7ffab559a640 1 -- 172.21.15.63:0/1877572979 >> v1:172.21.15.63:6789/0 conn(0x7ffab81430c0 legacy=0x7ffab814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:14:54.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.660+0000 7ffab559a640 10 monclient: _finish_hunting 0 2024-03-25T18:14:54.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.660+0000 7ffab559a640 1 monclient: found mon.noname-c 2024-03-25T18:14:54.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.660+0000 7ffab559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:54.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.660+0000 7ffab559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:14:54.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.660+0000 7ffab559a640 1 -- 172.21.15.63:0/1877572979 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab80b44c0 con 0x7ffab813f710 2024-03-25T18:14:54.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.660+0000 7ffab559a640 10 monclient: _finish_auth 0 2024-03-25T18:14:54.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.660+0000 7ffab559a640 10 monclient: _check_auth_tickets 2024-03-25T18:14:54.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.660+0000 7ffab559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:54.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.660+0000 7ffabc904640 5 monclient: authenticate success, global_id 4327 2024-03-25T18:14:54.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.660+0000 7ffabc904640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:54.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.662+0000 7ffab559a640 1 -- 172.21.15.63:0/1877572979 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ffaac001690 con 0x7ffab813f710 2024-03-25T18:14:54.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.662+0000 7ffab559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:54.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.662+0000 7ffab559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:54.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.662+0000 7ffab559a640 10 monclient: dump: 2024-03-25T18:14:54.663 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:54.663 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:54.663 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:54.663 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:54.664 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:54.664 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:54.664 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:54.664 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:54.664 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:54.664 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:54.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.662+0000 7ffab559a640 1 -- 172.21.15.63:0/1877572979 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffaac001960 con 0x7ffab813f710 2024-03-25T18:14:54.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.662+0000 7ffabc904640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:54.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.662+0000 7ffab559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:54.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.662+0000 7ffabc904640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:54.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.662+0000 7ffab559a640 1 -- 172.21.15.63:0/1877572979 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ffaac001cf0 con 0x7ffab813f710 2024-03-25T18:14:54.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.662+0000 7ffabc904640 4 set_mon_vals no callback set 2024-03-25T18:14:54.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.662+0000 7ffab559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:54.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.662+0000 7ffab559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:54.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.662+0000 7ffab559a640 10 monclient: dump: 2024-03-25T18:14:54.665 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:54.665 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:54.665 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:54.665 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:54.665 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:54.665 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:54.665 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:54.665 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:54.665 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:54.665 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:54.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.663+0000 7ffabc904640 10 monclient: shutdown 2024-03-25T18:14:54.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.663+0000 7ffabc904640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:54.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.663+0000 7ffabc904640 1 -- 172.21.15.63:0/1877572979 >> v1:172.21.15.81:6789/0 conn(0x7ffab813f710 legacy=0x7ffab81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:54.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.663+0000 7ffabc904640 1 -- 172.21.15.63:0/1877572979 shutdown_connections 2024-03-25T18:14:54.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.663+0000 7ffabc904640 1 -- 172.21.15.63:0/1877572979 >> 172.21.15.63:0/1877572979 conn(0x7ffab80b7ea0 msgr2=0x7ffab80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:54.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.663+0000 7ffabc904640 1 -- 172.21.15.63:0/1877572979 shutdown_connections 2024-03-25T18:14:54.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.663+0000 7ffabc904640 1 -- 172.21.15.63:0/1877572979 wait complete. 2024-03-25T18:14:54.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.664+0000 7ffabc904640 10 monclient: build_initial_monmap 2024-03-25T18:14:54.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.664+0000 7ffabc904640 10 monclient: monmap: 2024-03-25T18:14:54.668 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:14:54.668 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:54.668 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:14:54.668 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:14:54.668 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:54.668 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:14:54.668 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:14:54.668 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:14:54.668 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:14:54.668 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:54.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.664+0000 7ffabc904640 1 librados: starting msgr at 2024-03-25T18:14:54.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.664+0000 7ffabc904640 1 librados: starting objecter 2024-03-25T18:14:54.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.664+0000 7ffabc904640 1 Processor -- start 2024-03-25T18:14:54.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.664+0000 7ffabc904640 1 -- start start 2024-03-25T18:14:54.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.664+0000 7ffabc904640 1 librados: setting wanted keys 2024-03-25T18:14:54.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.664+0000 7ffabc904640 1 librados: calling monclient init 2024-03-25T18:14:54.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.664+0000 7ffabc904640 10 monclient: init 2024-03-25T18:14:54.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.665+0000 7ffabc904640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:54.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.665+0000 7ffabc904640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:54.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.665+0000 7ffabc904640 10 monclient(hunting): picked mon.noname-c con 0x7ffab80a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:14:54.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.665+0000 7ffabc904640 10 monclient(hunting): picked mon.noname-b con 0x7ffab813f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:14:54.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.665+0000 7ffabc904640 10 monclient(hunting): picked mon.noname-a con 0x7ffab81430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:14:54.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.665+0000 7ffabc904640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab80ab8e0 con 0x7ffab81430c0 2024-03-25T18:14:54.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.665+0000 7ffabc904640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab80abb90 con 0x7ffab813f710 2024-03-25T18:14:54.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.665+0000 7ffabc904640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab80abe70 con 0x7ffab80a98f0 2024-03-25T18:14:54.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.665+0000 7ffabc904640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:54.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.665+0000 7ffabc904640 10 monclient(hunting): authenticate will time out at 7588.851965s 2024-03-25T18:14:54.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.675+0000 7ffab5d9b640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ffab813f710 0x7ffab80ac300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:33908/0 (socket says 172.21.15.63:33908) 2024-03-25T18:14:54.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.675+0000 7ffab5d9b640 1 -- 172.21.15.63:0/3023449795 learned_addr learned my addr 172.21.15.63:0/3023449795 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:14:54.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.677+0000 7ffaa6ffd640 10 client.?.objecter ms_handle_connect 0x7ffab813f710 2024-03-25T18:14:54.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.677+0000 7ffaa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:54.680 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.679+0000 7ffaa6ffd640 10 client.?.objecter ms_handle_connect 0x7ffab80a98f0 2024-03-25T18:14:54.680 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.679+0000 7ffaa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:54.683 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.682+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584918705 0 0) 0x7ffab80abb90 con 0x7ffab813f710 2024-03-25T18:14:54.683 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.682+0000 7ffaa6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:54.683 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.682+0000 7ffaa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:54.683 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.682+0000 7ffaa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:54.683 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.682+0000 7ffaa6ffd640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:14:54.683 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.682+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa98004eb0 con 0x7ffab813f710 2024-03-25T18:14:54.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.689+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2664996971 0 0) 0x7ffab80abe70 con 0x7ffab80a98f0 2024-03-25T18:14:54.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.689+0000 7ffaa6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:54.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.689+0000 7ffaa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:54.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.689+0000 7ffaa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:54.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.689+0000 7ffaa6ffd640 10 monclient(hunting): my global_id is 4325 2024-03-25T18:14:54.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.689+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab80abb90 con 0x7ffab80a98f0 2024-03-25T18:14:54.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.698+0000 7ffaa6ffd640 10 client.?.objecter ms_handle_connect 0x7ffab81430c0 2024-03-25T18:14:54.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.698+0000 7ffaa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:54.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.707+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3307138469 0 0) 0x7ffa98004eb0 con 0x7ffab813f710 2024-03-25T18:14:54.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.707+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffab80abe70 con 0x7ffab813f710 2024-03-25T18:14:54.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.707+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ffaac001650 con 0x7ffab813f710 2024-03-25T18:14:54.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.707+0000 7ffaa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:54.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.707+0000 7ffaa6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:54.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.707+0000 7ffaa6ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:54.709 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:54.709 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:54.709 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:54.709 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:54.709 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:54.709 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:54.709 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:54.709 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:54.709 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:54.709 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:54.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.707+0000 7ffaa6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:54.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.707+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2228870590 0 0) 0x7ffab80abb90 con 0x7ffab80a98f0 2024-03-25T18:14:54.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.707+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa98004eb0 con 0x7ffab80a98f0 2024-03-25T18:14:54.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.707+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ffaa8001510 con 0x7ffab80a98f0 2024-03-25T18:14:54.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.707+0000 7ffaa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:54.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.707+0000 7ffaa6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:54.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.707+0000 7ffaa6ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:54.710 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:54.710 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:54.710 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:54.710 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:54.710 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:54.710 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:54.710 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:54.710 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:54.710 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:54.711 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:54.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.710+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615735525 0 0) 0x7ffab80ab8e0 con 0x7ffab81430c0 2024-03-25T18:14:54.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.710+0000 7ffaa6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:54.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.710+0000 7ffaa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:54.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.710+0000 7ffaa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:54.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.710+0000 7ffaa6ffd640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:14:54.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.710+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab80abb90 con 0x7ffab81430c0 2024-03-25T18:14:54.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.725+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1654402302 0 0) 0x7ffab80abe70 con 0x7ffab813f710 2024-03-25T18:14:54.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 >> v1:172.21.15.63:6790/0 conn(0x7ffab80a98f0 legacy=0x7ffab81386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:54.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 >> v1:172.21.15.63:6789/0 conn(0x7ffab81430c0 legacy=0x7ffab8136350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:54.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffaa6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:54.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffaa6ffd640 1 monclient: found mon.b 2024-03-25T18:14:54.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffaa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:54.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffaa6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:54.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab8138e80 con 0x7ffab813f710 2024-03-25T18:14:54.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffaa6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:54.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffaa6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:54.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffaa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:54.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffabc904640 5 monclient: authenticate success, global_id 4330 2024-03-25T18:14:54.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffabc904640 10 monclient: _renew_subs 2024-03-25T18:14:54.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffabc904640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:54.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffabc904640 1 -- 172.21.15.63:0/3023449795 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab8137b30 con 0x7ffab813f710 2024-03-25T18:14:54.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffabc904640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:54.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffabc904640 10 monclient: _renew_subs 2024-03-25T18:14:54.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffabc904640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:54.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffabc904640 1 -- 172.21.15.63:0/3023449795 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab81e3480 con 0x7ffab813f710 2024-03-25T18:14:54.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.726+0000 7ffabc904640 1 librados: init done 2024-03-25T18:14:54.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.727+0000 7ffabc904640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:54.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.727+0000 7ffabc904640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:54.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.727+0000 7ffabc904640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:54.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.727+0000 7ffabc904640 1 -- 172.21.15.63:0/3023449795 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab80b3ef0 con 0x7ffab813f710 2024-03-25T18:14:54.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.943+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffaac004c70 con 0x7ffab813f710 2024-03-25T18:14:54.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.943+0000 7ffaa6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:54.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.943+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ffaac005080 con 0x7ffab813f710 2024-03-25T18:14:54.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.943+0000 7ffaa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:54.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.943+0000 7ffaa6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:54.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.943+0000 7ffaa6ffd640 10 monclient: dump: 2024-03-25T18:14:54.944 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:14:54.944 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:14:54.945 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:54.945 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:14:54.945 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:54.945 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:14:54.945 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:14:54.945 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:14:54.945 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:14:54.945 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:14:54.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:54.943+0000 7ffab4d99640 4 set_mon_vals no callback set 2024-03-25T18:14:55.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.006+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7ffaac0051e0 con 0x7ffab813f710 2024-03-25T18:14:55.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.007+0000 7ffab659c640 10 monclient: get_auth_request con 0x7ffa9806df40 auth_method 0 2024-03-25T18:14:55.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.007+0000 7ffaa6ffd640 10 client.4330.objecter ms_handle_connect 0x7ffa9806df40 2024-03-25T18:14:55.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.015+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7ffaac054a30 con 0x7ffab813f710 2024-03-25T18:14:55.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.015+0000 7ffaa6ffd640 10 client.4330.objecter ms_dispatch 0x7ffab8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:55.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.015+0000 7ffaa6ffd640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:55.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.015+0000 7ffaa6ffd640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:55.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.015+0000 7ffaa6ffd640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-25T18:14:55.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.018+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffaac050540 con 0x7ffab813f710 2024-03-25T18:14:55.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.018+0000 7ffaa6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:55.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.018+0000 7ffaa6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:55.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.271+0000 7ffabc904640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:55.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.271+0000 7ffabc904640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:55.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.271+0000 7ffabc904640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:14:55.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.271+0000 7ffabc904640 1 -- 172.21.15.63:0/3023449795 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffab80627b0 con 0x7ffab813f710 2024-03-25T18:14:55.301 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.300+0000 7ffaa6ffd640 1 -- 172.21.15.63:0/3023449795 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7ffaac052470 con 0x7ffab813f710 2024-03-25T18:14:55.301 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.300+0000 7ffaa6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:55.301 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.300+0000 7ffaa6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:55.301 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:14:55.301 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:55.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.302+0000 7ffabc904640 10 librados: watch_flush enter 2024-03-25T18:14:55.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.302+0000 7ffabc904640 10 librados: watch_flush exit 2024-03-25T18:14:55.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.302+0000 7ffabc904640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-25T18:14:55.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.302+0000 7ffabc904640 10 client.4330.objecter successfully canceled tick 2024-03-25T18:14:55.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.302+0000 7ffabc904640 1 -- 172.21.15.63:0/3023449795 >> v1:172.21.15.81:6800/3640340873 conn(0x7ffa9806df40 legacy=0x7ffa98070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:55.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.302+0000 7ffabc904640 10 monclient: shutdown 2024-03-25T18:14:55.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.302+0000 7ffabc904640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:55.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.303+0000 7ffabc904640 1 -- 172.21.15.63:0/3023449795 >> v1:172.21.15.81:6789/0 conn(0x7ffab813f710 legacy=0x7ffab80ac300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:55.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.303+0000 7ffabc904640 1 -- 172.21.15.63:0/3023449795 shutdown_connections 2024-03-25T18:14:55.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.303+0000 7ffabc904640 1 -- 172.21.15.63:0/3023449795 >> 172.21.15.63:0/3023449795 conn(0x7ffab80b7ea0 msgr2=0x7ffab8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:55.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.303+0000 7ffabc904640 1 -- 172.21.15.63:0/3023449795 shutdown_connections 2024-03-25T18:14:55.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.303+0000 7ffabc904640 1 -- 172.21.15.63:0/3023449795 wait complete. 2024-03-25T18:14:55.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:14:55.303+0000 7ffabc904640 1 librados: shutdown 2024-03-25T18:14:55.314 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:02.120 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:02.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.232+0000 7fd86bb8d640 10 monclient: get_monmap_and_config 2024-03-25T18:15:02.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.232+0000 7fd86bb8d640 10 monclient: build_initial_monmap 2024-03-25T18:15:02.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.232+0000 7fd86bb8d640 10 monclient: monmap: 2024-03-25T18:15:02.234 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:15:02.234 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:02.234 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:15:02.233937+0000 2024-03-25T18:15:02.234 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:15:02.233937+0000 2024-03-25T18:15:02.234 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:02.234 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:02.234 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:15:02.234 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:15:02.234 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:15:02.234 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:02.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.233+0000 7fd86bb8d640 1 Processor -- start 2024-03-25T18:15:02.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.233+0000 7fd86bb8d640 1 -- start start 2024-03-25T18:15:02.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.233+0000 7fd86bb8d640 10 monclient: init 2024-03-25T18:15:02.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.234+0000 7fd86bb8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:02.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.234+0000 7fd86bb8d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:02.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.234+0000 7fd86bb8d640 10 monclient(hunting): picked mon.noname-a con 0x7fd8640a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:15:02.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.234+0000 7fd86bb8d640 10 monclient(hunting): picked mon.noname-b con 0x7fd86413f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:15:02.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.234+0000 7fd86bb8d640 10 monclient(hunting): picked mon.noname-c con 0x7fd8641430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:15:02.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.234+0000 7fd86bb8d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd864142520 con 0x7fd8640a98f0 2024-03-25T18:15:02.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.234+0000 7fd86bb8d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8640b28a0 con 0x7fd8641430c0 2024-03-25T18:15:02.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.234+0000 7fd86bb8d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8640b2a30 con 0x7fd86413f710 2024-03-25T18:15:02.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.234+0000 7fd86bb8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:02.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.234+0000 7fd86bb8d640 10 monclient(hunting): authenticate will time out at 7596.420901s 2024-03-25T18:15:02.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.245+0000 7fd869128640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fd86413f710 0x7fd8641429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:50458/0 (socket says 172.21.15.63:50458) 2024-03-25T18:15:02.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.245+0000 7fd869128640 1 -- 172.21.15.63:0/703279171 learned_addr learned my addr 172.21.15.63:0/703279171 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:15:02.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.259+0000 7fd868927640 1 -- 172.21.15.63:0/703279171 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306707721 0 0) 0x7fd8640b28a0 con 0x7fd8641430c0 2024-03-25T18:15:02.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.259+0000 7fd868927640 10 monclient(hunting): _negotiate 2024-03-25T18:15:02.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.259+0000 7fd868927640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:02.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.259+0000 7fd868927640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:02.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.260+0000 7fd868927640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:15:02.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.260+0000 7fd868927640 1 -- 172.21.15.63:0/703279171 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd850003520 con 0x7fd8641430c0 2024-03-25T18:15:02.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.265+0000 7fd868927640 1 -- 172.21.15.63:0/703279171 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965831177 0 0) 0x7fd8640b2a30 con 0x7fd86413f710 2024-03-25T18:15:02.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.265+0000 7fd868927640 10 monclient(hunting): _negotiate 2024-03-25T18:15:02.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.265+0000 7fd868927640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:02.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.265+0000 7fd868927640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:02.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.265+0000 7fd868927640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:15:02.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.265+0000 7fd868927640 1 -- 172.21.15.63:0/703279171 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8640b28a0 con 0x7fd86413f710 2024-03-25T18:15:02.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.278+0000 7fd868927640 1 -- 172.21.15.63:0/703279171 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812412449 0 0) 0x7fd864142520 con 0x7fd8640a98f0 2024-03-25T18:15:02.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.278+0000 7fd868927640 10 monclient(hunting): _negotiate 2024-03-25T18:15:02.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.278+0000 7fd868927640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:02.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.278+0000 7fd868927640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:02.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.278+0000 7fd868927640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:15:02.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.278+0000 7fd868927640 1 -- 172.21.15.63:0/703279171 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8640b2a30 con 0x7fd8640a98f0 2024-03-25T18:15:02.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd868927640 1 -- 172.21.15.63:0/703279171 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3635800483 0 0) 0x7fd850003520 con 0x7fd8641430c0 2024-03-25T18:15:02.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd868927640 1 -- 172.21.15.63:0/703279171 >> v1:172.21.15.63:6790/0 conn(0x7fd86413f710 legacy=0x7fd8641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:02.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd868927640 1 -- 172.21.15.63:0/703279171 >> v1:172.21.15.63:6789/0 conn(0x7fd8640a98f0 legacy=0x7fd8640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:02.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd868927640 10 monclient: _finish_hunting 0 2024-03-25T18:15:02.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd868927640 1 monclient: found mon.noname-c 2024-03-25T18:15:02.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd868927640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:02.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd868927640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:15:02.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd868927640 1 -- 172.21.15.63:0/703279171 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8640b44c0 con 0x7fd8641430c0 2024-03-25T18:15:02.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd868927640 10 monclient: _finish_auth 0 2024-03-25T18:15:02.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd868927640 10 monclient: _check_auth_tickets 2024-03-25T18:15:02.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd868927640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:02.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd868927640 1 -- 172.21.15.63:0/703279171 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fd85c001a10 con 0x7fd8641430c0 2024-03-25T18:15:02.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd868927640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:02.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd868927640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:02.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd868927640 10 monclient: dump: 2024-03-25T18:15:02.285 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:02.285 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:02.285 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:02.285 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:02.285 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:02.285 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:02.285 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:02.285 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:02.286 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:02.286 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:02.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd86bb8d640 5 monclient: authenticate success, global_id 4333 2024-03-25T18:15:02.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.283+0000 7fd86bb8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:02.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.308+0000 7fd868927640 1 -- 172.21.15.63:0/703279171 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd85c001b70 con 0x7fd8641430c0 2024-03-25T18:15:02.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.308+0000 7fd868927640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:02.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.308+0000 7fd868927640 1 -- 172.21.15.63:0/703279171 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fd85c001f80 con 0x7fd8641430c0 2024-03-25T18:15:02.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.308+0000 7fd86bb8d640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:02.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.308+0000 7fd86bb8d640 4 set_mon_vals no callback set 2024-03-25T18:15:02.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.308+0000 7fd868927640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:02.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.308+0000 7fd868927640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:02.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.308+0000 7fd868927640 10 monclient: dump: 2024-03-25T18:15:02.310 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:02.310 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:02.310 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:02.310 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:02.310 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:02.310 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:02.310 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:02.310 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:02.310 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:02.310 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:02.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.309+0000 7fd86bb8d640 10 monclient: shutdown 2024-03-25T18:15:02.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.309+0000 7fd86bb8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:02.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.309+0000 7fd86bb8d640 1 -- 172.21.15.63:0/703279171 >> v1:172.21.15.81:6789/0 conn(0x7fd8641430c0 legacy=0x7fd86414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:02.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.309+0000 7fd86bb8d640 1 -- 172.21.15.63:0/703279171 shutdown_connections 2024-03-25T18:15:02.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.309+0000 7fd86bb8d640 1 -- 172.21.15.63:0/703279171 >> 172.21.15.63:0/703279171 conn(0x7fd8640b7ea0 msgr2=0x7fd8640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:02.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.309+0000 7fd86bb8d640 1 -- 172.21.15.63:0/703279171 shutdown_connections 2024-03-25T18:15:02.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.309+0000 7fd86bb8d640 1 -- 172.21.15.63:0/703279171 wait complete. 2024-03-25T18:15:02.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.310+0000 7fd86bb8d640 10 monclient: build_initial_monmap 2024-03-25T18:15:02.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.310+0000 7fd86bb8d640 10 monclient: monmap: 2024-03-25T18:15:02.313 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:15:02.313 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:02.313 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:15:02.313 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:15:02.313 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:02.313 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:15:02.313 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:15:02.313 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:15:02.313 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:15:02.313 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:02.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.310+0000 7fd86bb8d640 1 librados: starting msgr at 2024-03-25T18:15:02.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.310+0000 7fd86bb8d640 1 librados: starting objecter 2024-03-25T18:15:02.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.310+0000 7fd86bb8d640 1 Processor -- start 2024-03-25T18:15:02.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.310+0000 7fd86bb8d640 1 -- start start 2024-03-25T18:15:02.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.310+0000 7fd86bb8d640 1 librados: setting wanted keys 2024-03-25T18:15:02.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.310+0000 7fd86bb8d640 1 librados: calling monclient init 2024-03-25T18:15:02.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.310+0000 7fd86bb8d640 10 monclient: init 2024-03-25T18:15:02.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.311+0000 7fd86bb8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:02.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.311+0000 7fd86bb8d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:02.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.311+0000 7fd86bb8d640 10 monclient(hunting): picked mon.noname-a con 0x7fd8640a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:15:02.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.311+0000 7fd86bb8d640 10 monclient(hunting): picked mon.noname-c con 0x7fd86413f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:15:02.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.311+0000 7fd86bb8d640 10 monclient(hunting): picked mon.noname-b con 0x7fd8641430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:15:02.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.311+0000 7fd86bb8d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8641daca0 con 0x7fd8640a98f0 2024-03-25T18:15:02.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.311+0000 7fd86bb8d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd864138ab0 con 0x7fd8641430c0 2024-03-25T18:15:02.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.311+0000 7fd86bb8d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8640aba90 con 0x7fd86413f710 2024-03-25T18:15:02.315 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.311+0000 7fd86bb8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:02.315 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.311+0000 7fd86bb8d640 10 monclient(hunting): authenticate will time out at 7596.497789s 2024-03-25T18:15:02.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.316+0000 7fd86a12a640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fd8641430c0 0x7fd8641d47b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:33386/0 (socket says 172.21.15.63:33386) 2024-03-25T18:15:02.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.316+0000 7fd86a12a640 1 -- 172.21.15.63:0/86882143 learned_addr learned my addr 172.21.15.63:0/86882143 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:15:02.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.320+0000 7fd85a7fc640 10 client.?.objecter ms_handle_connect 0x7fd8641430c0 2024-03-25T18:15:02.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.320+0000 7fd85a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:02.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.331+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621928195 0 0) 0x7fd864138ab0 con 0x7fd8641430c0 2024-03-25T18:15:02.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.331+0000 7fd85a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:02.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.331+0000 7fd85a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:02.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.331+0000 7fd85a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:02.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.331+0000 7fd85a7fc640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:15:02.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.331+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd848004eb0 con 0x7fd8641430c0 2024-03-25T18:15:02.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.333+0000 7fd85a7fc640 10 client.?.objecter ms_handle_connect 0x7fd86413f710 2024-03-25T18:15:02.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.333+0000 7fd85a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:02.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.338+0000 7fd85a7fc640 10 client.?.objecter ms_handle_connect 0x7fd8640a98f0 2024-03-25T18:15:02.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.338+0000 7fd85a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:02.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.339+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281207421 0 0) 0x7fd8640aba90 con 0x7fd86413f710 2024-03-25T18:15:02.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.339+0000 7fd85a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:02.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.339+0000 7fd85a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:02.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.339+0000 7fd85a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:02.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.339+0000 7fd85a7fc640 10 monclient(hunting): my global_id is 4331 2024-03-25T18:15:02.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.339+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd864138ab0 con 0x7fd86413f710 2024-03-25T18:15:02.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.345+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1801009555 0 0) 0x7fd848004eb0 con 0x7fd8641430c0 2024-03-25T18:15:02.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.345+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8640aba90 con 0x7fd8641430c0 2024-03-25T18:15:02.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.345+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fd85c001f40 con 0x7fd8641430c0 2024-03-25T18:15:02.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.345+0000 7fd85a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:02.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.345+0000 7fd85a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:02.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.345+0000 7fd85a7fc640 10 monclient(hunting): dump: 2024-03-25T18:15:02.347 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:02.347 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:02.347 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:02.347 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:02.347 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:02.348 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:02.348 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:02.348 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:02.348 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:02.348 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:02.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.345+0000 7fd85a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:02.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.354+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 926319401 0 0) 0x7fd864138ab0 con 0x7fd86413f710 2024-03-25T18:15:02.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.354+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd848004eb0 con 0x7fd86413f710 2024-03-25T18:15:02.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.354+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fd854001970 con 0x7fd86413f710 2024-03-25T18:15:02.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.354+0000 7fd85a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:02.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.354+0000 7fd85a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:02.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.354+0000 7fd85a7fc640 10 monclient(hunting): dump: 2024-03-25T18:15:02.356 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:02.356 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:02.356 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:02.356 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:02.356 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:02.357 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:02.357 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:02.357 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:02.357 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:02.357 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:02.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.354+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392409843 0 0) 0x7fd8641daca0 con 0x7fd8640a98f0 2024-03-25T18:15:02.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.354+0000 7fd85a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:02.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.354+0000 7fd85a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:02.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.354+0000 7fd85a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:02.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.354+0000 7fd85a7fc640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:15:02.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.354+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd864138ab0 con 0x7fd8640a98f0 2024-03-25T18:15:02.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3037198648 0 0) 0x7fd848004eb0 con 0x7fd86413f710 2024-03-25T18:15:02.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 >> v1:172.21.15.81:6789/0 conn(0x7fd8641430c0 legacy=0x7fd8641d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:02.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 >> v1:172.21.15.63:6789/0 conn(0x7fd8640a98f0 legacy=0x7fd8641383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:02.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd85a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:02.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd85a7fc640 1 monclient: found mon.c 2024-03-25T18:15:02.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd85a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:02.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd85a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:15:02.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8641d41e0 con 0x7fd86413f710 2024-03-25T18:15:02.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd85a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:02.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd85a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:02.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd85a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:02.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd86bb8d640 5 monclient: authenticate success, global_id 4331 2024-03-25T18:15:02.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd86bb8d640 10 monclient: _renew_subs 2024-03-25T18:15:02.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd86bb8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:15:02.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd86bb8d640 1 -- 172.21.15.63:0/86882143 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8641db840 con 0x7fd86413f710 2024-03-25T18:15:02.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd86bb8d640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:02.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd86bb8d640 10 monclient: _renew_subs 2024-03-25T18:15:02.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd86bb8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:15:02.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd86bb8d640 1 -- 172.21.15.63:0/86882143 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd864138f00 con 0x7fd86413f710 2024-03-25T18:15:02.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.361+0000 7fd86bb8d640 1 librados: init done 2024-03-25T18:15:02.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.362+0000 7fd86bb8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:02.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.362+0000 7fd86bb8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:02.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.362+0000 7fd86bb8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:15:02.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.362+0000 7fd86bb8d640 1 -- 172.21.15.63:0/86882143 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8641dae50 con 0x7fd86413f710 2024-03-25T18:15:02.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.388+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8540047e0 con 0x7fd86413f710 2024-03-25T18:15:02.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.388+0000 7fd85a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:02.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.388+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fd854004950 con 0x7fd86413f710 2024-03-25T18:15:02.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.388+0000 7fd85a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:02.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.388+0000 7fd85a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:02.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.388+0000 7fd85a7fc640 10 monclient: dump: 2024-03-25T18:15:02.390 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:02.390 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:02.390 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:02.391 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:02.391 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:02.391 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:02.391 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:02.391 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:02.391 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:02.391 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:02.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.388+0000 7fd85bfff640 4 set_mon_vals no callback set 2024-03-25T18:15:02.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.389+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fd85401a920 con 0x7fd86413f710 2024-03-25T18:15:02.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.390+0000 7fd869929640 10 monclient: get_auth_request con 0x7fd84806df30 auth_method 0 2024-03-25T18:15:02.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.390+0000 7fd85a7fc640 10 client.4331.objecter ms_handle_connect 0x7fd84806df30 2024-03-25T18:15:02.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.676+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fd854054620 con 0x7fd86413f710 2024-03-25T18:15:02.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.676+0000 7fd85a7fc640 10 client.4331.objecter ms_dispatch 0x7fd864000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:02.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.676+0000 7fd85a7fc640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:02.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.676+0000 7fd85a7fc640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:02.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.676+0000 7fd85a7fc640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-25T18:15:02.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.676+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd85404ff60 con 0x7fd86413f710 2024-03-25T18:15:02.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.676+0000 7fd85a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:02.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.676+0000 7fd85a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:02.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.939+0000 7fd86bb8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:02.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.939+0000 7fd86bb8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:02.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.939+0000 7fd86bb8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:15:02.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.939+0000 7fd86bb8d640 1 -- 172.21.15.63:0/86882143 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8640627b0 con 0x7fd86413f710 2024-03-25T18:15:02.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.972+0000 7fd85a7fc640 1 -- 172.21.15.63:0/86882143 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fd854052060 con 0x7fd86413f710 2024-03-25T18:15:02.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.972+0000 7fd85a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:02.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.972+0000 7fd85a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:02.974 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:15:02.974 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:02.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.974+0000 7fd86bb8d640 10 librados: watch_flush enter 2024-03-25T18:15:02.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.975+0000 7fd86bb8d640 10 librados: watch_flush exit 2024-03-25T18:15:02.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.975+0000 7fd86bb8d640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-25T18:15:02.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.975+0000 7fd86bb8d640 10 client.4331.objecter successfully canceled tick 2024-03-25T18:15:02.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.975+0000 7fd86bb8d640 1 -- 172.21.15.63:0/86882143 >> v1:172.21.15.81:6800/3640340873 conn(0x7fd84806df30 legacy=0x7fd8480703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.975+0000 7fd86bb8d640 10 monclient: shutdown 2024-03-25T18:15:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.975+0000 7fd86bb8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.975+0000 7fd86bb8d640 1 -- 172.21.15.63:0/86882143 >> v1:172.21.15.63:6790/0 conn(0x7fd86413f710 legacy=0x7fd8641db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.975+0000 7fd86bb8d640 1 -- 172.21.15.63:0/86882143 shutdown_connections 2024-03-25T18:15:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.975+0000 7fd86bb8d640 1 -- 172.21.15.63:0/86882143 >> 172.21.15.63:0/86882143 conn(0x7fd8640b7ea0 msgr2=0x7fd864074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.975+0000 7fd86bb8d640 1 -- 172.21.15.63:0/86882143 shutdown_connections 2024-03-25T18:15:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.975+0000 7fd86bb8d640 1 -- 172.21.15.63:0/86882143 wait complete. 2024-03-25T18:15:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:02.975+0000 7fd86bb8d640 1 librados: shutdown 2024-03-25T18:15:02.987 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:09.794 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:09.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.907+0000 7f3e83893640 10 monclient: get_monmap_and_config 2024-03-25T18:15:09.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.907+0000 7f3e83893640 10 monclient: build_initial_monmap 2024-03-25T18:15:09.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.907+0000 7f3e83893640 10 monclient: monmap: 2024-03-25T18:15:09.909 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:15:09.909 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:09.909 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:15:09.908934+0000 2024-03-25T18:15:09.909 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:15:09.908934+0000 2024-03-25T18:15:09.909 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:09.909 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:09.909 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:15:09.909 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:15:09.909 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:15:09.909 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:09.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.908+0000 7f3e83893640 1 Processor -- start 2024-03-25T18:15:09.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.908+0000 7f3e83893640 1 -- start start 2024-03-25T18:15:09.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.908+0000 7f3e83893640 10 monclient: init 2024-03-25T18:15:09.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.909+0000 7f3e83893640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:09.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.909+0000 7f3e83893640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:09.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.909+0000 7f3e83893640 10 monclient(hunting): picked mon.noname-b con 0x7f3e7c0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:15:09.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.909+0000 7f3e83893640 10 monclient(hunting): picked mon.noname-c con 0x7f3e7c13f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:15:09.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.909+0000 7f3e83893640 10 monclient(hunting): picked mon.noname-a con 0x7f3e7c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:15:09.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.909+0000 7f3e83893640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7c142520 con 0x7f3e7c1430c0 2024-03-25T18:15:09.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.909+0000 7f3e83893640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7c0b28a0 con 0x7f3e7c13f710 2024-03-25T18:15:09.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.909+0000 7f3e83893640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7c0b2a30 con 0x7f3e7c0a98f0 2024-03-25T18:15:09.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.909+0000 7f3e83893640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:09.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.909+0000 7f3e83893640 10 monclient(hunting): authenticate will time out at 7604.095926s 2024-03-25T18:15:09.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.914+0000 7f3e80e2e640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f3e7c13f710 0x7f3e7c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:52764/0 (socket says 172.21.15.63:52764) 2024-03-25T18:15:09.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.914+0000 7f3e80e2e640 1 -- 172.21.15.63:0/3004778986 learned_addr learned my addr 172.21.15.63:0/3004778986 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:15:09.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.919+0000 7f3e73fff640 1 -- 172.21.15.63:0/3004778986 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378485584 0 0) 0x7f3e7c0b28a0 con 0x7f3e7c13f710 2024-03-25T18:15:09.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.919+0000 7f3e73fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:09.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.919+0000 7f3e73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:09.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.919+0000 7f3e73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:09.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.920+0000 7f3e73fff640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:15:09.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.920+0000 7f3e73fff640 1 -- 172.21.15.63:0/3004778986 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e68003520 con 0x7f3e7c13f710 2024-03-25T18:15:09.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.927+0000 7f3e73fff640 1 -- 172.21.15.63:0/3004778986 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646193944 0 0) 0x7f3e7c0b2a30 con 0x7f3e7c0a98f0 2024-03-25T18:15:09.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.927+0000 7f3e73fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:09.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.927+0000 7f3e73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:09.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.927+0000 7f3e73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:09.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.927+0000 7f3e73fff640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:15:09.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.927+0000 7f3e73fff640 1 -- 172.21.15.63:0/3004778986 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e7c0b28a0 con 0x7f3e7c0a98f0 2024-03-25T18:15:09.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e73fff640 1 -- 172.21.15.63:0/3004778986 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2064135619 0 0) 0x7f3e68003520 con 0x7f3e7c13f710 2024-03-25T18:15:09.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e73fff640 1 -- 172.21.15.63:0/3004778986 >> v1:172.21.15.63:6790/0 conn(0x7f3e7c0a98f0 legacy=0x7f3e7c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:09.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e73fff640 1 -- 172.21.15.63:0/3004778986 >> v1:172.21.15.63:6789/0 conn(0x7f3e7c1430c0 legacy=0x7f3e7c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:09.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e73fff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:09.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e73fff640 1 monclient: found mon.noname-c 2024-03-25T18:15:09.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:09.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e73fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:15:09.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e73fff640 1 -- 172.21.15.63:0/3004778986 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e7c0b44c0 con 0x7f3e7c13f710 2024-03-25T18:15:09.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e73fff640 10 monclient: _finish_auth 0 2024-03-25T18:15:09.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e73fff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:09.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:09.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e73fff640 1 -- 172.21.15.63:0/3004778986 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3e6c001690 con 0x7f3e7c13f710 2024-03-25T18:15:09.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e83893640 5 monclient: authenticate success, global_id 4339 2024-03-25T18:15:09.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e83893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:09.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:09.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e73fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:09.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e73fff640 10 monclient: dump: 2024-03-25T18:15:09.936 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:09.936 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:09.936 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:09.936 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:09.936 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:09.936 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:09.936 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:09.936 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:09.936 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:09.936 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:09.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.933+0000 7f3e83893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:09.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.950+0000 7f3e73fff640 1 -- 172.21.15.63:0/3004778986 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e6c0017f0 con 0x7f3e7c13f710 2024-03-25T18:15:09.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.950+0000 7f3e73fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:09.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.950+0000 7f3e73fff640 1 -- 172.21.15.63:0/3004778986 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3e6c001c00 con 0x7f3e7c13f710 2024-03-25T18:15:09.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.950+0000 7f3e73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:09.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.950+0000 7f3e73fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:09.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.950+0000 7f3e73fff640 10 monclient: dump: 2024-03-25T18:15:09.952 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:09.952 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:09.952 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:09.952 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:09.952 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:09.952 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:09.952 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:09.952 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:09.952 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:09.952 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:09.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.950+0000 7f3e83893640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:09.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.950+0000 7f3e83893640 4 set_mon_vals no callback set 2024-03-25T18:15:09.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.951+0000 7f3e83893640 10 monclient: shutdown 2024-03-25T18:15:09.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.951+0000 7f3e83893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:09.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.951+0000 7f3e83893640 1 -- 172.21.15.63:0/3004778986 >> v1:172.21.15.81:6789/0 conn(0x7f3e7c13f710 legacy=0x7f3e7c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:09.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.951+0000 7f3e83893640 1 -- 172.21.15.63:0/3004778986 shutdown_connections 2024-03-25T18:15:09.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.951+0000 7f3e83893640 1 -- 172.21.15.63:0/3004778986 >> 172.21.15.63:0/3004778986 conn(0x7f3e7c0b7ea0 msgr2=0x7f3e7c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:09.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.951+0000 7f3e83893640 1 -- 172.21.15.63:0/3004778986 shutdown_connections 2024-03-25T18:15:09.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.951+0000 7f3e83893640 1 -- 172.21.15.63:0/3004778986 wait complete. 2024-03-25T18:15:09.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.952+0000 7f3e83893640 10 monclient: build_initial_monmap 2024-03-25T18:15:09.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.952+0000 7f3e83893640 10 monclient: monmap: 2024-03-25T18:15:09.955 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:15:09.955 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:09.956 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:15:09.956 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:15:09.956 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:09.956 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:15:09.956 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:15:09.956 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:15:09.956 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:15:09.956 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:09.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.952+0000 7f3e83893640 1 librados: starting msgr at 2024-03-25T18:15:09.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.952+0000 7f3e83893640 1 librados: starting objecter 2024-03-25T18:15:09.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.952+0000 7f3e83893640 1 Processor -- start 2024-03-25T18:15:09.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.952+0000 7f3e83893640 1 -- start start 2024-03-25T18:15:09.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.952+0000 7f3e83893640 1 librados: setting wanted keys 2024-03-25T18:15:09.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.952+0000 7f3e83893640 1 librados: calling monclient init 2024-03-25T18:15:09.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.952+0000 7f3e83893640 10 monclient: init 2024-03-25T18:15:09.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.953+0000 7f3e83893640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:09.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.953+0000 7f3e83893640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:09.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.953+0000 7f3e83893640 10 monclient(hunting): picked mon.noname-c con 0x7f3e7c0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:15:09.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.953+0000 7f3e83893640 10 monclient(hunting): picked mon.noname-a con 0x7f3e7c13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:15:09.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.953+0000 7f3e83893640 10 monclient(hunting): picked mon.noname-b con 0x7f3e7c1430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:15:09.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.953+0000 7f3e83893640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7c1d77a0 con 0x7f3e7c13f710 2024-03-25T18:15:09.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.953+0000 7f3e83893640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7c138e40 con 0x7f3e7c1430c0 2024-03-25T18:15:09.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.953+0000 7f3e83893640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7c0aba30 con 0x7f3e7c0a98f0 2024-03-25T18:15:09.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.953+0000 7f3e83893640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:09.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.953+0000 7f3e83893640 10 monclient(hunting): authenticate will time out at 7604.140049s 2024-03-25T18:15:09.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.963+0000 7f3e81e30640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f3e7c1430c0 0x7f3e7c134110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:52804/0 (socket says 172.21.15.63:52804) 2024-03-25T18:15:09.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.963+0000 7f3e81e30640 1 -- 172.21.15.63:0/3108271236 learned_addr learned my addr 172.21.15.63:0/3108271236 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:15:09.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.968+0000 7f3e71ffb640 10 client.?.objecter ms_handle_connect 0x7f3e7c1430c0 2024-03-25T18:15:09.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.968+0000 7f3e71ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:09.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.975+0000 7f3e71ffb640 10 client.?.objecter ms_handle_connect 0x7f3e7c13f710 2024-03-25T18:15:09.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.975+0000 7f3e71ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:09.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.984+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669365929 0 0) 0x7f3e7c138e40 con 0x7f3e7c1430c0 2024-03-25T18:15:09.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.984+0000 7f3e71ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:09.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.984+0000 7f3e71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:09.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.984+0000 7f3e71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:09.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.984+0000 7f3e71ffb640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:15:09.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.984+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e60004eb0 con 0x7f3e7c1430c0 2024-03-25T18:15:09.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.992+0000 7f3e71ffb640 10 client.?.objecter ms_handle_connect 0x7f3e7c0a98f0 2024-03-25T18:15:09.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.992+0000 7f3e71ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:09.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.993+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 549522393 0 0) 0x7f3e60004eb0 con 0x7f3e7c1430c0 2024-03-25T18:15:09.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.993+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e7c138e40 con 0x7f3e7c1430c0 2024-03-25T18:15:09.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.995+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3e74001d70 con 0x7f3e7c1430c0 2024-03-25T18:15:09.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.995+0000 7f3e71ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:09.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.995+0000 7f3e71ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:09.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.995+0000 7f3e71ffb640 10 monclient(hunting): dump: 2024-03-25T18:15:09.996 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:09.996 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:09.996 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:09.996 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:09.996 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:09.996 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:09.996 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:09.996 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:09.996 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:09.996 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:09.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.995+0000 7f3e71ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:09.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.997+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350704794 0 0) 0x7f3e7c1d77a0 con 0x7f3e7c13f710 2024-03-25T18:15:09.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.997+0000 7f3e71ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:09.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.997+0000 7f3e71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:09.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.997+0000 7f3e71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:09.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.997+0000 7f3e71ffb640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:15:09.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:09.997+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e60004eb0 con 0x7f3e7c13f710 2024-03-25T18:15:10.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2676580975 0 0) 0x7f3e7c138e40 con 0x7f3e7c1430c0 2024-03-25T18:15:10.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 >> v1:172.21.15.63:6790/0 conn(0x7f3e7c0a98f0 legacy=0x7f3e7c138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:10.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 >> v1:172.21.15.63:6789/0 conn(0x7f3e7c13f710 legacy=0x7f3e7c1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:10.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e71ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:15:10.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e71ffb640 1 monclient: found mon.b 2024-03-25T18:15:10.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:10.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e71ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:10.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e7c1d8340 con 0x7f3e7c1430c0 2024-03-25T18:15:10.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e71ffb640 10 monclient: _finish_auth 0 2024-03-25T18:15:10.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e71ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:15:10.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:10.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e83893640 5 monclient: authenticate success, global_id 4342 2024-03-25T18:15:10.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e83893640 10 monclient: _renew_subs 2024-03-25T18:15:10.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e83893640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:10.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e83893640 1 -- 172.21.15.63:0/3108271236 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e7c134820 con 0x7f3e7c1430c0 2024-03-25T18:15:10.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e83893640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:10.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e83893640 10 monclient: _renew_subs 2024-03-25T18:15:10.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e83893640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:10.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e83893640 1 -- 172.21.15.63:0/3108271236 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e7c1380c0 con 0x7f3e7c1430c0 2024-03-25T18:15:10.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.002+0000 7f3e83893640 1 librados: init done 2024-03-25T18:15:10.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.003+0000 7f3e83893640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:10.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.003+0000 7f3e83893640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:10.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.003+0000 7f3e83893640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:10.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.003+0000 7f3e83893640 1 -- 172.21.15.63:0/3108271236 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e7c0627b0 con 0x7f3e7c1430c0 2024-03-25T18:15:10.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.276+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e74003a10 con 0x7f3e7c1430c0 2024-03-25T18:15:10.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.276+0000 7f3e71ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:10.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.276+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3e74003e20 con 0x7f3e7c1430c0 2024-03-25T18:15:10.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.276+0000 7f3e71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:10.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.276+0000 7f3e71ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:10.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.276+0000 7f3e71ffb640 10 monclient: dump: 2024-03-25T18:15:10.279 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:10.279 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:10.279 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:10.280 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:10.280 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:10.280 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:10.280 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:10.280 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:10.280 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:10.280 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:10.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.276+0000 7f3e737fe640 4 set_mon_vals no callback set 2024-03-25T18:15:10.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.277+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f3e74004040 con 0x7f3e7c1430c0 2024-03-25T18:15:10.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.278+0000 7f3e8162f640 10 monclient: get_auth_request con 0x7f3e6006dd60 auth_method 0 2024-03-25T18:15:10.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.278+0000 7f3e71ffb640 10 client.4342.objecter ms_handle_connect 0x7f3e6006dd60 2024-03-25T18:15:10.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.281+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f3e74054be0 con 0x7f3e7c1430c0 2024-03-25T18:15:10.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.281+0000 7f3e71ffb640 10 client.4342.objecter ms_dispatch 0x7f3e7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:10.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.281+0000 7f3e71ffb640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:10.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.281+0000 7f3e71ffb640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:10.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.281+0000 7f3e71ffb640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-25T18:15:10.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.284+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3e740506f0 con 0x7f3e7c1430c0 2024-03-25T18:15:10.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.284+0000 7f3e71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:10.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.284+0000 7f3e71ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:10.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.534+0000 7f3e83893640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:10.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.534+0000 7f3e83893640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:10.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.534+0000 7f3e83893640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:10.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.534+0000 7f3e83893640 1 -- 172.21.15.63:0/3108271236 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e7c14e8b0 con 0x7f3e7c1430c0 2024-03-25T18:15:10.617 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.616+0000 7f3e71ffb640 1 -- 172.21.15.63:0/3108271236 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f3e74052620 con 0x7f3e7c1430c0 2024-03-25T18:15:10.617 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.616+0000 7f3e71ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:10.617 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.616+0000 7f3e71ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:10.617 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:15:10.618 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:10.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.618+0000 7f3e83893640 10 librados: watch_flush enter 2024-03-25T18:15:10.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.619+0000 7f3e83893640 10 librados: watch_flush exit 2024-03-25T18:15:10.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.619+0000 7f3e83893640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-25T18:15:10.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.619+0000 7f3e83893640 10 client.4342.objecter successfully canceled tick 2024-03-25T18:15:10.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.619+0000 7f3e83893640 1 -- 172.21.15.63:0/3108271236 >> v1:172.21.15.81:6800/3640340873 conn(0x7f3e6006dd60 legacy=0x7f3e60070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:10.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.619+0000 7f3e83893640 10 monclient: shutdown 2024-03-25T18:15:10.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.619+0000 7f3e83893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:10.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.619+0000 7f3e83893640 1 -- 172.21.15.63:0/3108271236 >> v1:172.21.15.81:6789/0 conn(0x7f3e7c1430c0 legacy=0x7f3e7c134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:10.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.619+0000 7f3e83893640 1 -- 172.21.15.63:0/3108271236 shutdown_connections 2024-03-25T18:15:10.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.619+0000 7f3e83893640 1 -- 172.21.15.63:0/3108271236 >> 172.21.15.63:0/3108271236 conn(0x7f3e7c0b7ea0 msgr2=0x7f3e7c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:10.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.619+0000 7f3e83893640 1 -- 172.21.15.63:0/3108271236 shutdown_connections 2024-03-25T18:15:10.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.619+0000 7f3e83893640 1 -- 172.21.15.63:0/3108271236 wait complete. 2024-03-25T18:15:10.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:10.619+0000 7f3e83893640 1 librados: shutdown 2024-03-25T18:15:10.632 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:13.412 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:55.105 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:13.417 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:55.110 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:13.442 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:55.134 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:13.469 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:55.162 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:13.470 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:02:55.131 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:13.481 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:55.173 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:13.486 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:02:55.147 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:13.512 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:02:55.172 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:13.517 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:55.209 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:17.439 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:17.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.547+0000 7fb2c0f5d640 10 monclient: get_monmap_and_config 2024-03-25T18:15:17.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.547+0000 7fb2c0f5d640 10 monclient: build_initial_monmap 2024-03-25T18:15:17.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.547+0000 7fb2c0f5d640 10 monclient: monmap: 2024-03-25T18:15:17.548 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:15:17.548 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:17.548 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:15:17.548793+0000 2024-03-25T18:15:17.549 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:15:17.548793+0000 2024-03-25T18:15:17.549 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:17.549 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:17.549 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:15:17.549 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:15:17.549 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:15:17.549 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:17.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.548+0000 7fb2c0f5d640 1 Processor -- start 2024-03-25T18:15:17.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.548+0000 7fb2c0f5d640 1 -- start start 2024-03-25T18:15:17.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.548+0000 7fb2c0f5d640 10 monclient: init 2024-03-25T18:15:17.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.548+0000 7fb2c0f5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:17.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.548+0000 7fb2c0f5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:17.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.548+0000 7fb2c0f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2bc0b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T18:15:17.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.548+0000 7fb2c0f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2bc0b1730 addr v1:172.21.15.63:6789/0 2024-03-25T18:15:17.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.549+0000 7fb2c0f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2bc14be90 addr v1:172.21.15.81:6789/0 2024-03-25T18:15:17.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.549+0000 7fb2c0f5d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc0af940 con 0x7fb2bc0b1730 2024-03-25T18:15:17.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.549+0000 7fb2c0f5d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc0b0a70 con 0x7fb2bc14be90 2024-03-25T18:15:17.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.549+0000 7fb2c0f5d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc0b0d50 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.549+0000 7fb2c0f5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:17.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.549+0000 7fb2c0f5d640 10 monclient(hunting): authenticate will time out at 7611.735328s 2024-03-25T18:15:17.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.549+0000 7fb2ba59c640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fb2bc0b4d60 0x7fb2bc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:34934/0 (socket says 172.21.15.63:34934) 2024-03-25T18:15:17.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.549+0000 7fb2ba59c640 1 -- 172.21.15.63:0/164182889 learned_addr learned my addr 172.21.15.63:0/164182889 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:15:17.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.590+0000 7fb2b9d9b640 1 -- 172.21.15.63:0/164182889 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000051405 0 0) 0x7fb2bc0b0a70 con 0x7fb2bc14be90 2024-03-25T18:15:17.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.590+0000 7fb2b9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:17.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.590+0000 7fb2b9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:17.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.590+0000 7fb2b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:17.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.590+0000 7fb2b9d9b640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:15:17.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.590+0000 7fb2b9d9b640 1 -- 172.21.15.63:0/164182889 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a8003520 con 0x7fb2bc14be90 2024-03-25T18:15:17.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2b9d9b640 1 -- 172.21.15.63:0/164182889 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1379499741 0 0) 0x7fb2a8003520 con 0x7fb2bc14be90 2024-03-25T18:15:17.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2b9d9b640 1 -- 172.21.15.63:0/164182889 >> v1:172.21.15.63:6790/0 conn(0x7fb2bc0b4d60 legacy=0x7fb2bc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:17.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2b9d9b640 1 -- 172.21.15.63:0/164182889 >> v1:172.21.15.63:6789/0 conn(0x7fb2bc0b1730 legacy=0x7fb2bc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:17.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2b9d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:15:17.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2b9d9b640 1 monclient: found mon.noname-c 2024-03-25T18:15:17.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2b9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:17.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2b9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:15:17.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2b9d9b640 1 -- 172.21.15.63:0/164182889 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc0b1030 con 0x7fb2bc14be90 2024-03-25T18:15:17.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2b9d9b640 10 monclient: _finish_auth 0 2024-03-25T18:15:17.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2b9d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:15:17.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2b9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:17.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2b9d9b640 1 -- 172.21.15.63:0/164182889 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fb2b4001a10 con 0x7fb2bc14be90 2024-03-25T18:15:17.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:17.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2b9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:17.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2b9d9b640 10 monclient: dump: 2024-03-25T18:15:17.611 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:17.611 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:17.611 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:17.611 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:17.611 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:17.612 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:17.612 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:17.612 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:17.612 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:17.612 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:17.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2c0f5d640 5 monclient: authenticate success, global_id 4345 2024-03-25T18:15:17.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.609+0000 7fb2c0f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:17.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.627+0000 7fb2b9d9b640 1 -- 172.21.15.63:0/164182889 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2b4001b70 con 0x7fb2bc14be90 2024-03-25T18:15:17.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.627+0000 7fb2b9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:17.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.627+0000 7fb2b9d9b640 1 -- 172.21.15.63:0/164182889 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fb2b4001fc0 con 0x7fb2bc14be90 2024-03-25T18:15:17.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.627+0000 7fb2c0f5d640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:17.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.627+0000 7fb2c0f5d640 4 set_mon_vals no callback set 2024-03-25T18:15:17.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.627+0000 7fb2b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:17.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.627+0000 7fb2b9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:17.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.627+0000 7fb2b9d9b640 10 monclient: dump: 2024-03-25T18:15:17.628 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:17.628 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:17.628 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:17.628 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:17.628 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:17.629 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:17.629 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:17.629 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:17.629 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:17.629 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:17.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.628+0000 7fb2c0f5d640 10 monclient: shutdown 2024-03-25T18:15:17.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.628+0000 7fb2c0f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:17.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.628+0000 7fb2c0f5d640 1 -- 172.21.15.63:0/164182889 >> v1:172.21.15.81:6789/0 conn(0x7fb2bc14be90 legacy=0x7fb2bc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:17.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.628+0000 7fb2c0f5d640 1 -- 172.21.15.63:0/164182889 shutdown_connections 2024-03-25T18:15:17.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.628+0000 7fb2c0f5d640 1 -- 172.21.15.63:0/164182889 >> 172.21.15.63:0/164182889 conn(0x7fb2bc0617a0 msgr2=0x7fb2bc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:17.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.628+0000 7fb2c0f5d640 1 -- 172.21.15.63:0/164182889 shutdown_connections 2024-03-25T18:15:17.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.628+0000 7fb2c0f5d640 1 -- 172.21.15.63:0/164182889 wait complete. 2024-03-25T18:15:17.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.629+0000 7fb2c0f5d640 10 monclient: build_initial_monmap 2024-03-25T18:15:17.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.629+0000 7fb2c0f5d640 10 monclient: monmap: 2024-03-25T18:15:17.631 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:15:17.631 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:17.631 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:15:17.631 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:15:17.631 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:17.631 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:15:17.631 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:15:17.631 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:15:17.631 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:15:17.632 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:17.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.629+0000 7fb2c0f5d640 1 librados: starting msgr at 2024-03-25T18:15:17.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.629+0000 7fb2c0f5d640 1 librados: starting objecter 2024-03-25T18:15:17.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.629+0000 7fb2c0f5d640 1 Processor -- start 2024-03-25T18:15:17.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.629+0000 7fb2c0f5d640 1 -- start start 2024-03-25T18:15:17.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.629+0000 7fb2c0f5d640 1 librados: setting wanted keys 2024-03-25T18:15:17.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.629+0000 7fb2c0f5d640 1 librados: calling monclient init 2024-03-25T18:15:17.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.629+0000 7fb2c0f5d640 10 monclient: init 2024-03-25T18:15:17.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.630+0000 7fb2c0f5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:17.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.630+0000 7fb2c0f5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:17.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.630+0000 7fb2c0f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2bc0b1730 addr v1:172.21.15.63:6789/0 2024-03-25T18:15:17.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.630+0000 7fb2c0f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2bc0b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T18:15:17.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.630+0000 7fb2c0f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2bc14be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:15:17.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.630+0000 7fb2c0f5d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc140930 con 0x7fb2bc0b1730 2024-03-25T18:15:17.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.630+0000 7fb2c0f5d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc145870 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.630+0000 7fb2c0f5d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc144dd0 con 0x7fb2bc14be90 2024-03-25T18:15:17.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.630+0000 7fb2c0f5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:17.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.630+0000 7fb2c0f5d640 10 monclient(hunting): authenticate will time out at 7611.817012s 2024-03-25T18:15:17.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.640+0000 7fb2b3fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb2bc0b4d60 0x7fb2bc1319c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:57568/0 (socket says 172.21.15.63:57568) 2024-03-25T18:15:17.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.640+0000 7fb2b3fff640 1 -- 172.21.15.63:0/2642576848 learned_addr learned my addr 172.21.15.63:0/2642576848 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:15:17.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.640+0000 7fb2b2ffd640 10 client.?.objecter ms_handle_connect 0x7fb2bc0b4d60 2024-03-25T18:15:17.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.640+0000 7fb2b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:17.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.643+0000 7fb2b2ffd640 1 -- 172.21.15.63:0/2642576848 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775762926 0 0) 0x7fb2bc145870 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.643+0000 7fb2b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:17.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.643+0000 7fb2b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:17.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.643+0000 7fb2b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:17.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.643+0000 7fb2b2ffd640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:15:17.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.643+0000 7fb2b2ffd640 1 -- 172.21.15.63:0/2642576848 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb29c004eb0 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.652+0000 7fb2b2ffd640 1 -- 172.21.15.63:0/2642576848 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 450004699 0 0) 0x7fb29c004eb0 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.652+0000 7fb2b2ffd640 1 -- 172.21.15.63:0/2642576848 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2bc145870 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.652+0000 7fb2b2ffd640 1 -- 172.21.15.63:0/2642576848 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fb2a4001a30 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.652+0000 7fb2b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:17.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.652+0000 7fb2b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:17.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.652+0000 7fb2b2ffd640 10 monclient(hunting): dump: 2024-03-25T18:15:17.653 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:17.653 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:17.653 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:17.654 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:17.654 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:17.654 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:17.654 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:17.654 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:17.654 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:17.654 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:17.654 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.652+0000 7fb2b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:17.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.660+0000 7fb2b2ffd640 10 client.?.objecter ms_handle_connect 0x7fb2bc0b1730 2024-03-25T18:15:17.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.660+0000 7fb2b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:17.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2b2ffd640 1 -- 172.21.15.63:0/2642576848 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4125824126 0 0) 0x7fb2bc145870 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2b2ffd640 1 -- 172.21.15.63:0/2642576848 >> v1:172.21.15.63:6790/0 conn(0x7fb2bc14be90 legacy=0x7fb2bc1320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:17.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2b2ffd640 1 -- 172.21.15.63:0/2642576848 >> v1:172.21.15.63:6789/0 conn(0x7fb2bc0b1730 legacy=0x7fb2bc1daca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:17.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:17.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2b2ffd640 1 monclient: found mon.b 2024-03-25T18:15:17.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:17.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2b2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:17.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2b2ffd640 1 -- 172.21.15.63:0/2642576848 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc144f60 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2b2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:17.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2b2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:17.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:17.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2c0f5d640 5 monclient: authenticate success, global_id 4348 2024-03-25T18:15:17.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2c0f5d640 10 monclient: _renew_subs 2024-03-25T18:15:17.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2c0f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:17.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2c0f5d640 1 -- 172.21.15.63:0/2642576848 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2bc140380 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2c0f5d640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:17.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2c0f5d640 10 monclient: _renew_subs 2024-03-25T18:15:17.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2c0f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:17.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2c0f5d640 1 -- 172.21.15.63:0/2642576848 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2bc13e1f0 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.661+0000 7fb2c0f5d640 1 librados: init done 2024-03-25T18:15:17.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.662+0000 7fb2c0f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:17.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.662+0000 7fb2c0f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:17.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.662+0000 7fb2c0f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:17.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.662+0000 7fb2c0f5d640 1 -- 172.21.15.63:0/2642576848 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2bc140ac0 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.949+0000 7fb2b2ffd640 1 -- 172.21.15.63:0/2642576848 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2a4001d30 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.949+0000 7fb2b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:17.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.949+0000 7fb2b2ffd640 1 -- 172.21.15.63:0/2642576848 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fb2a4003a60 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.949+0000 7fb2b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:17.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.949+0000 7fb2b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:17.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.949+0000 7fb2b2ffd640 10 monclient: dump: 2024-03-25T18:15:17.951 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:17.951 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:17.951 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:17.951 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:17.951 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:17.951 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:17.951 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:17.951 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:17.951 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:17.951 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:17.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.949+0000 7fb2b959a640 4 set_mon_vals no callback set 2024-03-25T18:15:17.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.950+0000 7fb2b2ffd640 1 -- 172.21.15.63:0/2642576848 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fb2a4003c80 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.950+0000 7fb2ba59c640 10 monclient: get_auth_request con 0x7fb29c06dbd0 auth_method 0 2024-03-25T18:15:17.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.951+0000 7fb2b2ffd640 10 client.4348.objecter ms_handle_connect 0x7fb29c06dbd0 2024-03-25T18:15:17.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.955+0000 7fb2b2ffd640 1 -- 172.21.15.63:0/2642576848 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fb2a4054970 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.955+0000 7fb2b2ffd640 10 client.4348.objecter ms_dispatch 0x7fb2bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:17.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.955+0000 7fb2b2ffd640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:17.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.955+0000 7fb2b2ffd640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:17.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.955+0000 7fb2b2ffd640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-25T18:15:17.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.956+0000 7fb2b2ffd640 1 -- 172.21.15.63:0/2642576848 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2a4050480 con 0x7fb2bc0b4d60 2024-03-25T18:15:17.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.956+0000 7fb2b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:17.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:17.956+0000 7fb2b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:18.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.212+0000 7fb2c0f5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:18.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.212+0000 7fb2c0f5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:18.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.212+0000 7fb2c0f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:18.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.212+0000 7fb2c0f5d640 1 -- 172.21.15.63:0/2642576848 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2bc140720 con 0x7fb2bc0b4d60 2024-03-25T18:15:18.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.231+0000 7fb2b2ffd640 1 -- 172.21.15.63:0/2642576848 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fb2a40523b0 con 0x7fb2bc0b4d60 2024-03-25T18:15:18.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.231+0000 7fb2b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:18.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.231+0000 7fb2b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:18.233 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:15:18.233 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:18.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.234+0000 7fb2c0f5d640 10 librados: watch_flush enter 2024-03-25T18:15:18.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.234+0000 7fb2c0f5d640 10 librados: watch_flush exit 2024-03-25T18:15:18.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.234+0000 7fb2c0f5d640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-25T18:15:18.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.234+0000 7fb2c0f5d640 10 client.4348.objecter successfully canceled tick 2024-03-25T18:15:18.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.234+0000 7fb2c0f5d640 1 -- 172.21.15.63:0/2642576848 >> v1:172.21.15.81:6800/3640340873 conn(0x7fb29c06dbd0 legacy=0x7fb29c070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:18.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.234+0000 7fb2c0f5d640 10 monclient: shutdown 2024-03-25T18:15:18.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.234+0000 7fb2c0f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:18.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.234+0000 7fb2c0f5d640 1 -- 172.21.15.63:0/2642576848 >> v1:172.21.15.81:6789/0 conn(0x7fb2bc0b4d60 legacy=0x7fb2bc1319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:18.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.234+0000 7fb2c0f5d640 1 -- 172.21.15.63:0/2642576848 shutdown_connections 2024-03-25T18:15:18.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.234+0000 7fb2c0f5d640 1 -- 172.21.15.63:0/2642576848 >> 172.21.15.63:0/2642576848 conn(0x7fb2bc0617a0 msgr2=0x7fb2bc1384b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:18.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.234+0000 7fb2c0f5d640 1 -- 172.21.15.63:0/2642576848 shutdown_connections 2024-03-25T18:15:18.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.234+0000 7fb2c0f5d640 1 -- 172.21.15.63:0/2642576848 wait complete. 2024-03-25T18:15:18.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:18.234+0000 7fb2c0f5d640 1 librados: shutdown 2024-03-25T18:15:18.246 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:25.054 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:25.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.174+0000 7f6365b1c640 10 monclient: get_monmap_and_config 2024-03-25T18:15:25.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.174+0000 7f6365b1c640 10 monclient: build_initial_monmap 2024-03-25T18:15:25.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.175+0000 7f6365b1c640 10 monclient: monmap: 2024-03-25T18:15:25.175 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:15:25.175 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:25.175 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:15:25.175135+0000 2024-03-25T18:15:25.175 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:15:25.175135+0000 2024-03-25T18:15:25.175 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:25.175 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:25.175 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:15:25.175 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:15:25.175 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:15:25.175 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:25.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.175+0000 7f6365b1c640 1 Processor -- start 2024-03-25T18:15:25.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.175+0000 7f6365b1c640 1 -- start start 2024-03-25T18:15:25.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.175+0000 7f6365b1c640 10 monclient: init 2024-03-25T18:15:25.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.176+0000 7f6365b1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:25.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.176+0000 7f6365b1c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:25.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.176+0000 7f6365b1c640 10 monclient(hunting): picked mon.noname-b con 0x7f63600a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:15:25.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.176+0000 7f6365b1c640 10 monclient(hunting): picked mon.noname-c con 0x7f636013f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:15:25.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.176+0000 7f6365b1c640 10 monclient(hunting): picked mon.noname-a con 0x7f63601430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:15:25.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.176+0000 7f6365b1c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6360142520 con 0x7f63601430c0 2024-03-25T18:15:25.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.176+0000 7f6365b1c640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63600b28a0 con 0x7f636013f710 2024-03-25T18:15:25.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.176+0000 7f6365b1c640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63600b2a30 con 0x7f63600a98f0 2024-03-25T18:15:25.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.176+0000 7f6365b1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:25.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.176+0000 7f6365b1c640 10 monclient(hunting): authenticate will time out at 7619.361879s 2024-03-25T18:15:25.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.186+0000 7f635f7fe640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f63600a98f0 0x7f63600a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:51864/0 (socket says 172.21.15.63:51864) 2024-03-25T18:15:25.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.187+0000 7f635f7fe640 1 -- 172.21.15.63:0/3594644687 learned_addr learned my addr 172.21.15.63:0/3594644687 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:15:25.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.196+0000 7f635e7fc640 1 -- 172.21.15.63:0/3594644687 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151201468 0 0) 0x7f63600b2a30 con 0x7f63600a98f0 2024-03-25T18:15:25.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.196+0000 7f635e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:25.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.196+0000 7f635e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:25.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.196+0000 7f635e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:25.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.196+0000 7f635e7fc640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:15:25.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.196+0000 7f635e7fc640 1 -- 172.21.15.63:0/3594644687 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6348003520 con 0x7f63600a98f0 2024-03-25T18:15:25.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.202+0000 7f635e7fc640 1 -- 172.21.15.63:0/3594644687 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002526409 0 0) 0x7f6360142520 con 0x7f63601430c0 2024-03-25T18:15:25.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.202+0000 7f635e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:25.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.202+0000 7f635e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:25.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.202+0000 7f635e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:25.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.202+0000 7f635e7fc640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:15:25.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.202+0000 7f635e7fc640 1 -- 172.21.15.63:0/3594644687 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63600b2a30 con 0x7f63601430c0 2024-03-25T18:15:25.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f635e7fc640 1 -- 172.21.15.63:0/3594644687 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 804810602 0 0) 0x7f6348003520 con 0x7f63600a98f0 2024-03-25T18:15:25.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f635e7fc640 1 -- 172.21.15.63:0/3594644687 >> v1:172.21.15.81:6789/0 conn(0x7f636013f710 legacy=0x7f63601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:25.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f635e7fc640 1 -- 172.21.15.63:0/3594644687 >> v1:172.21.15.63:6789/0 conn(0x7f63601430c0 legacy=0x7f636014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:25.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f635e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:25.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f635e7fc640 1 monclient: found mon.noname-b 2024-03-25T18:15:25.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f635e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:25.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f635e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:15:25.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f635e7fc640 1 -- 172.21.15.63:0/3594644687 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63600b44c0 con 0x7f63600a98f0 2024-03-25T18:15:25.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f635e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:25.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f635e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:25.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f635e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:25.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f635e7fc640 1 -- 172.21.15.63:0/3594644687 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6350001690 con 0x7f63600a98f0 2024-03-25T18:15:25.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f635e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:25.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f635e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:25.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f635e7fc640 10 monclient: dump: 2024-03-25T18:15:25.213 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:25.213 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:25.213 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:25.213 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:25.213 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:25.213 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:25.213 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:25.213 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:25.213 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:25.213 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:25.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f6365b1c640 5 monclient: authenticate success, global_id 4343 2024-03-25T18:15:25.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.211+0000 7f6365b1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:25.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.215+0000 7f635e7fc640 1 -- 172.21.15.63:0/3594644687 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63500017f0 con 0x7f63600a98f0 2024-03-25T18:15:25.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.215+0000 7f635e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:25.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.215+0000 7f6365b1c640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:25.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.215+0000 7f6365b1c640 4 set_mon_vals no callback set 2024-03-25T18:15:25.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.216+0000 7f6365b1c640 10 monclient: shutdown 2024-03-25T18:15:25.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.216+0000 7f6365b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:25.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.216+0000 7f6365b1c640 1 -- 172.21.15.63:0/3594644687 >> v1:172.21.15.63:6790/0 conn(0x7f63600a98f0 legacy=0x7f63600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:25.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.216+0000 7f6365b1c640 1 -- 172.21.15.63:0/3594644687 shutdown_connections 2024-03-25T18:15:25.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.216+0000 7f6365b1c640 1 -- 172.21.15.63:0/3594644687 >> 172.21.15.63:0/3594644687 conn(0x7f63600b7ea0 msgr2=0x7f63600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:25.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.216+0000 7f6365b1c640 1 -- 172.21.15.63:0/3594644687 shutdown_connections 2024-03-25T18:15:25.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.216+0000 7f6365b1c640 1 -- 172.21.15.63:0/3594644687 wait complete. 2024-03-25T18:15:25.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.217+0000 7f6365b1c640 10 monclient: build_initial_monmap 2024-03-25T18:15:25.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.217+0000 7f6365b1c640 10 monclient: monmap: 2024-03-25T18:15:25.218 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:15:25.218 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:25.218 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:15:25.218 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:15:25.218 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:25.218 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:15:25.218 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:15:25.218 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:15:25.218 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:15:25.218 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:25.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.218+0000 7f6365b1c640 1 librados: starting msgr at 2024-03-25T18:15:25.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.218+0000 7f6365b1c640 1 librados: starting objecter 2024-03-25T18:15:25.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.218+0000 7f6365b1c640 1 Processor -- start 2024-03-25T18:15:25.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.218+0000 7f6365b1c640 1 -- start start 2024-03-25T18:15:25.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.218+0000 7f6365b1c640 1 librados: setting wanted keys 2024-03-25T18:15:25.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.218+0000 7f6365b1c640 1 librados: calling monclient init 2024-03-25T18:15:25.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.218+0000 7f6365b1c640 10 monclient: init 2024-03-25T18:15:25.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.218+0000 7f6365b1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:25.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.218+0000 7f6365b1c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:25.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.218+0000 7f6365b1c640 10 monclient(hunting): picked mon.noname-a con 0x7f63600a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:15:25.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.218+0000 7f6365b1c640 10 monclient(hunting): picked mon.noname-b con 0x7f636013f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:15:25.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.219+0000 7f6365b1c640 10 monclient(hunting): picked mon.noname-c con 0x7f63601430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:15:25.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.219+0000 7f6365b1c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6360138700 con 0x7f63600a98f0 2024-03-25T18:15:25.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.219+0000 7f6365b1c640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6360138930 con 0x7f636013f710 2024-03-25T18:15:25.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.219+0000 7f6365b1c640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6360138c10 con 0x7f63601430c0 2024-03-25T18:15:25.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.219+0000 7f6365b1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:25.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.219+0000 7f6365b1c640 10 monclient(hunting): authenticate will time out at 7619.404435s 2024-03-25T18:15:25.227 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.228+0000 7f635ffff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f63601430c0 0x7f63601e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:51882/0 (socket says 172.21.15.63:51882) 2024-03-25T18:15:25.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.228+0000 7f635ffff640 1 -- 172.21.15.63:0/1734338317 learned_addr learned my addr 172.21.15.63:0/1734338317 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:15:25.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.228+0000 7f6347fff640 10 client.?.objecter ms_handle_connect 0x7f63601430c0 2024-03-25T18:15:25.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.228+0000 7f6347fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:25.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.240+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113401311 0 0) 0x7f6360138c10 con 0x7f63601430c0 2024-03-25T18:15:25.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.240+0000 7f6347fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:25.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.240+0000 7f6347fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:25.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.240+0000 7f6347fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:25.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.240+0000 7f6347fff640 10 monclient(hunting): my global_id is 4346 2024-03-25T18:15:25.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.240+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f634c004eb0 con 0x7f63601430c0 2024-03-25T18:15:25.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.240+0000 7f6347fff640 10 client.?.objecter ms_handle_connect 0x7f636013f710 2024-03-25T18:15:25.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.240+0000 7f6347fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:25.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.242+0000 7f6347fff640 10 client.?.objecter ms_handle_connect 0x7f63600a98f0 2024-03-25T18:15:25.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.242+0000 7f6347fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:25.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.250+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626827935 0 0) 0x7f6360138930 con 0x7f636013f710 2024-03-25T18:15:25.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.250+0000 7f6347fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:25.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.250+0000 7f6347fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:25.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.250+0000 7f6347fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:25.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.250+0000 7f6347fff640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:15:25.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.250+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6360138c10 con 0x7f636013f710 2024-03-25T18:15:25.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.257+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2455979985 0 0) 0x7f634c004eb0 con 0x7f63601430c0 2024-03-25T18:15:25.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.257+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6360138930 con 0x7f63601430c0 2024-03-25T18:15:25.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.257+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6358001c70 con 0x7f63601430c0 2024-03-25T18:15:25.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.257+0000 7f6347fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:25.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.257+0000 7f6347fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:25.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.257+0000 7f6347fff640 10 monclient(hunting): dump: 2024-03-25T18:15:25.258 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:25.258 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:25.258 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:25.258 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:25.258 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:25.258 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:25.258 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:25.258 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:25.258 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:25.258 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:25.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.257+0000 7f6347fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:25.265 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.266+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314338570 0 0) 0x7f6360138700 con 0x7f63600a98f0 2024-03-25T18:15:25.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.266+0000 7f6347fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:25.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.266+0000 7f6347fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:25.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.266+0000 7f6347fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:25.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.266+0000 7f6347fff640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:15:25.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.266+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f634c004eb0 con 0x7f63600a98f0 2024-03-25T18:15:25.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 940523 0 0) 0x7f6360138930 con 0x7f63601430c0 2024-03-25T18:15:25.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 >> v1:172.21.15.81:6789/0 conn(0x7f636013f710 legacy=0x7f6360137590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:25.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 >> v1:172.21.15.63:6789/0 conn(0x7f63600a98f0 legacy=0x7f63600b70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:25.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6347fff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:25.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6347fff640 1 monclient: found mon.c 2024-03-25T18:15:25.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6347fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:25.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6347fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:15:25.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6360138ef0 con 0x7f63601430c0 2024-03-25T18:15:25.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6347fff640 10 monclient: _finish_auth 0 2024-03-25T18:15:25.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6347fff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:25.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6347fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:25.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6365b1c640 5 monclient: authenticate success, global_id 4346 2024-03-25T18:15:25.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6365b1c640 10 monclient: _renew_subs 2024-03-25T18:15:25.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6365b1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:15:25.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6365b1c640 1 -- 172.21.15.63:0/1734338317 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6360136b70 con 0x7f63601430c0 2024-03-25T18:15:25.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6365b1c640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:25.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6365b1c640 10 monclient: _renew_subs 2024-03-25T18:15:25.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6365b1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:15:25.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6365b1c640 1 -- 172.21.15.63:0/1734338317 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6360137e80 con 0x7f63601430c0 2024-03-25T18:15:25.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.267+0000 7f6365b1c640 1 librados: init done 2024-03-25T18:15:25.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.270+0000 7f6365b1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:25.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.270+0000 7f6365b1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:25.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.270+0000 7f6365b1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:15:25.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.270+0000 7f6365b1c640 1 -- 172.21.15.63:0/1734338317 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6360139990 con 0x7f63601430c0 2024-03-25T18:15:25.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.583+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6358003990 con 0x7f63601430c0 2024-03-25T18:15:25.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.583+0000 7f6347fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:25.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.583+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6358003d60 con 0x7f63601430c0 2024-03-25T18:15:25.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.583+0000 7f6347fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:25.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.583+0000 7f6347fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:25.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.583+0000 7f6347fff640 10 monclient: dump: 2024-03-25T18:15:25.583 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:25.583 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:25.583 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:25.583 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:25.584 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:25.584 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:25.584 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:25.584 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:25.584 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:25.584 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:25.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.583+0000 7f635dffb640 4 set_mon_vals no callback set 2024-03-25T18:15:25.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.584+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f6358003f80 con 0x7f63601430c0 2024-03-25T18:15:25.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.584+0000 7f635f7fe640 10 monclient: get_auth_request con 0x7f634c06de00 auth_method 0 2024-03-25T18:15:25.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.585+0000 7f6347fff640 10 client.4346.objecter ms_handle_connect 0x7f634c06de00 2024-03-25T18:15:25.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.590+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f6358054ad0 con 0x7f63601430c0 2024-03-25T18:15:25.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.590+0000 7f6347fff640 10 client.4346.objecter ms_dispatch 0x7f6360000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:25.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.590+0000 7f6347fff640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:25.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.590+0000 7f6347fff640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:25.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.590+0000 7f6347fff640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-25T18:15:25.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.590+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f63580505e0 con 0x7f63601430c0 2024-03-25T18:15:25.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.590+0000 7f6347fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:25.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.590+0000 7f6347fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:25.845 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.845+0000 7f6365b1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:25.846 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.845+0000 7f6365b1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:25.846 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.845+0000 7f6365b1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:15:25.846 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.845+0000 7f6365b1c640 1 -- 172.21.15.63:0/1734338317 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63600627b0 con 0x7f63601430c0 2024-03-25T18:15:25.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.872+0000 7f6347fff640 1 -- 172.21.15.63:0/1734338317 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f6358052510 con 0x7f63601430c0 2024-03-25T18:15:25.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.872+0000 7f6347fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:25.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.872+0000 7f6347fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:25.872 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:15:25.873 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:25.875 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.874+0000 7f6365b1c640 10 librados: watch_flush enter 2024-03-25T18:15:25.875 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.874+0000 7f6365b1c640 10 librados: watch_flush exit 2024-03-25T18:15:25.875 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.874+0000 7f6365b1c640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-25T18:15:25.875 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.874+0000 7f6365b1c640 10 client.4346.objecter successfully canceled tick 2024-03-25T18:15:25.875 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.874+0000 7f6365b1c640 1 -- 172.21.15.63:0/1734338317 >> v1:172.21.15.81:6800/3640340873 conn(0x7f634c06de00 legacy=0x7f634c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:25.875 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.874+0000 7f6365b1c640 10 monclient: shutdown 2024-03-25T18:15:25.875 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.874+0000 7f6365b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:25.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.874+0000 7f6365b1c640 1 -- 172.21.15.63:0/1734338317 >> v1:172.21.15.63:6790/0 conn(0x7f63601430c0 legacy=0x7f63601e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:25.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.874+0000 7f6365b1c640 1 -- 172.21.15.63:0/1734338317 shutdown_connections 2024-03-25T18:15:25.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.874+0000 7f6365b1c640 1 -- 172.21.15.63:0/1734338317 >> 172.21.15.63:0/1734338317 conn(0x7f63600b7ea0 msgr2=0x7f636013edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:25.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.874+0000 7f6365b1c640 1 -- 172.21.15.63:0/1734338317 shutdown_connections 2024-03-25T18:15:25.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.874+0000 7f6365b1c640 1 -- 172.21.15.63:0/1734338317 wait complete. 2024-03-25T18:15:25.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:25.874+0000 7f6365b1c640 1 librados: shutdown 2024-03-25T18:15:25.886 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:32.693 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:32.811 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.810+0000 7fc7fd433640 10 monclient: get_monmap_and_config 2024-03-25T18:15:32.811 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.810+0000 7fc7fd433640 10 monclient: build_initial_monmap 2024-03-25T18:15:32.811 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.810+0000 7fc7fd433640 10 monclient: monmap: 2024-03-25T18:15:32.811 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:15:32.811 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:32.811 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:15:32.811229+0000 2024-03-25T18:15:32.811 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:15:32.811229+0000 2024-03-25T18:15:32.811 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:32.811 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:32.811 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:15:32.811 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:15:32.811 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:15:32.812 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:32.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.810+0000 7fc7fd433640 1 Processor -- start 2024-03-25T18:15:32.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.810+0000 7fc7fd433640 1 -- start start 2024-03-25T18:15:32.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.810+0000 7fc7fd433640 10 monclient: init 2024-03-25T18:15:32.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.811+0000 7fc7fd433640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:32.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.811+0000 7fc7fd433640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:32.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.811+0000 7fc7fd433640 10 monclient(hunting): picked mon.noname-b con 0x7fc7f80b1ef0 addr v1:172.21.15.63:6790/0 2024-03-25T18:15:32.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.811+0000 7fc7fd433640 10 monclient(hunting): picked mon.noname-c con 0x7fc7f81473a0 addr v1:172.21.15.81:6789/0 2024-03-25T18:15:32.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.811+0000 7fc7fd433640 10 monclient(hunting): picked mon.noname-a con 0x7fc7f8150070 addr v1:172.21.15.63:6789/0 2024-03-25T18:15:32.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.811+0000 7fc7fd433640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f80b2570 con 0x7fc7f8150070 2024-03-25T18:15:32.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.811+0000 7fc7fd433640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f8146c20 con 0x7fc7f81473a0 2024-03-25T18:15:32.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.811+0000 7fc7fd433640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f8146e80 con 0x7fc7f80b1ef0 2024-03-25T18:15:32.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.811+0000 7fc7fd433640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:32.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.811+0000 7fc7fd433640 10 monclient(hunting): authenticate will time out at 7626.998065s 2024-03-25T18:15:32.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.816+0000 7fc7f67fc640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc7f81473a0 0x7fc7f8143b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:52966/0 (socket says 172.21.15.63:52966) 2024-03-25T18:15:32.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.816+0000 7fc7f67fc640 1 -- 172.21.15.63:0/1720441472 learned_addr learned my addr 172.21.15.63:0/1720441472 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:15:32.832 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.831+0000 7fc7f5ffb640 1 -- 172.21.15.63:0/1720441472 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376716415 0 0) 0x7fc7f8146c20 con 0x7fc7f81473a0 2024-03-25T18:15:32.832 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.831+0000 7fc7f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:32.833 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.831+0000 7fc7f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:32.833 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.831+0000 7fc7f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:32.833 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.831+0000 7fc7f5ffb640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:15:32.833 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.831+0000 7fc7f5ffb640 1 -- 172.21.15.63:0/1720441472 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e0003520 con 0x7fc7f81473a0 2024-03-25T18:15:32.833 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.832+0000 7fc7f5ffb640 1 -- 172.21.15.63:0/1720441472 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319706933 0 0) 0x7fc7f80b2570 con 0x7fc7f8150070 2024-03-25T18:15:32.833 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.832+0000 7fc7f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:32.833 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.832+0000 7fc7f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:32.834 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.832+0000 7fc7f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:32.834 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.832+0000 7fc7f5ffb640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:15:32.834 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.832+0000 7fc7f5ffb640 1 -- 172.21.15.63:0/1720441472 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f8146c20 con 0x7fc7f8150070 2024-03-25T18:15:32.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7f5ffb640 1 -- 172.21.15.63:0/1720441472 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3545865012 0 0) 0x7fc7e0003520 con 0x7fc7f81473a0 2024-03-25T18:15:32.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7f5ffb640 1 -- 172.21.15.63:0/1720441472 >> v1:172.21.15.63:6790/0 conn(0x7fc7f80b1ef0 legacy=0x7fc7f80b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:32.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7f5ffb640 1 -- 172.21.15.63:0/1720441472 >> v1:172.21.15.63:6789/0 conn(0x7fc7f8150070 legacy=0x7fc7f8152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:32.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7f5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:15:32.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7f5ffb640 1 monclient: found mon.noname-c 2024-03-25T18:15:32.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:32.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7f5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:15:32.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7f5ffb640 1 -- 172.21.15.63:0/1720441472 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f806e1b0 con 0x7fc7f81473a0 2024-03-25T18:15:32.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7f5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:15:32.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7f5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:15:32.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:32.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7f5ffb640 1 -- 172.21.15.63:0/1720441472 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc7ec001690 con 0x7fc7f81473a0 2024-03-25T18:15:32.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:32.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7f5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:32.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7f5ffb640 10 monclient: dump: 2024-03-25T18:15:32.840 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:32.840 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:32.840 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:32.840 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:32.840 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:32.840 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:32.840 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:32.840 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:32.840 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:32.840 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:32.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7fd433640 5 monclient: authenticate success, global_id 4357 2024-03-25T18:15:32.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.838+0000 7fc7fd433640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:32.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.860+0000 7fc7f5ffb640 1 -- 172.21.15.63:0/1720441472 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7ec0017f0 con 0x7fc7f81473a0 2024-03-25T18:15:32.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.860+0000 7fc7f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:32.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.860+0000 7fc7f5ffb640 1 -- 172.21.15.63:0/1720441472 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc7ec001c00 con 0x7fc7f81473a0 2024-03-25T18:15:32.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.860+0000 7fc7fd433640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:32.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.860+0000 7fc7fd433640 4 set_mon_vals no callback set 2024-03-25T18:15:32.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.860+0000 7fc7f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:32.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.860+0000 7fc7f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:32.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.860+0000 7fc7f5ffb640 10 monclient: dump: 2024-03-25T18:15:32.861 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:32.861 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:32.862 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:32.862 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:32.862 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:32.862 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:32.862 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:32.862 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:32.862 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:32.862 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:32.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.861+0000 7fc7fd433640 10 monclient: shutdown 2024-03-25T18:15:32.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.861+0000 7fc7fd433640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:32.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.861+0000 7fc7fd433640 1 -- 172.21.15.63:0/1720441472 >> v1:172.21.15.81:6789/0 conn(0x7fc7f81473a0 legacy=0x7fc7f8143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:32.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.861+0000 7fc7fd433640 1 -- 172.21.15.63:0/1720441472 shutdown_connections 2024-03-25T18:15:32.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.861+0000 7fc7fd433640 1 -- 172.21.15.63:0/1720441472 >> 172.21.15.63:0/1720441472 conn(0x7fc7f80b7ea0 msgr2=0x7fc7f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:32.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.861+0000 7fc7fd433640 1 -- 172.21.15.63:0/1720441472 shutdown_connections 2024-03-25T18:15:32.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.861+0000 7fc7fd433640 1 -- 172.21.15.63:0/1720441472 wait complete. 2024-03-25T18:15:32.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.861+0000 7fc7fd433640 10 monclient: build_initial_monmap 2024-03-25T18:15:32.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.861+0000 7fc7fd433640 10 monclient: monmap: 2024-03-25T18:15:32.864 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:15:32.865 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:32.865 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:15:32.865 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:15:32.865 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:32.865 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:15:32.865 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:15:32.865 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:15:32.865 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:15:32.865 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:32.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.862+0000 7fc7fd433640 1 librados: starting msgr at 2024-03-25T18:15:32.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.862+0000 7fc7fd433640 1 librados: starting objecter 2024-03-25T18:15:32.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.862+0000 7fc7fd433640 1 Processor -- start 2024-03-25T18:15:32.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.862+0000 7fc7fd433640 1 -- start start 2024-03-25T18:15:32.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.862+0000 7fc7fd433640 1 librados: setting wanted keys 2024-03-25T18:15:32.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.862+0000 7fc7fd433640 1 librados: calling monclient init 2024-03-25T18:15:32.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.862+0000 7fc7fd433640 10 monclient: init 2024-03-25T18:15:32.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.863+0000 7fc7fd433640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:32.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.863+0000 7fc7fd433640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:32.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.863+0000 7fc7fd433640 10 monclient(hunting): picked mon.noname-c con 0x7fc7f80b1ef0 addr v1:172.21.15.63:6790/0 2024-03-25T18:15:32.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.863+0000 7fc7fd433640 10 monclient(hunting): picked mon.noname-b con 0x7fc7f81473a0 addr v1:172.21.15.81:6789/0 2024-03-25T18:15:32.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.863+0000 7fc7fd433640 10 monclient(hunting): picked mon.noname-a con 0x7fc7f8150070 addr v1:172.21.15.63:6789/0 2024-03-25T18:15:32.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.863+0000 7fc7fd433640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f81dbaa0 con 0x7fc7f8150070 2024-03-25T18:15:32.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.863+0000 7fc7fd433640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f8070280 con 0x7fc7f81473a0 2024-03-25T18:15:32.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.863+0000 7fc7fd433640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f8070410 con 0x7fc7f80b1ef0 2024-03-25T18:15:32.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.863+0000 7fc7fd433640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:32.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.863+0000 7fc7fd433640 10 monclient(hunting): authenticate will time out at 7627.049575s 2024-03-25T18:15:32.873 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.872+0000 7fc7f77fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fc7f8150070 0x7fc7f81458e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:44708/0 (socket says 172.21.15.63:44708) 2024-03-25T18:15:32.873 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.872+0000 7fc7f77fe640 1 -- 172.21.15.63:0/3128021875 learned_addr learned my addr 172.21.15.63:0/3128021875 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:15:32.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.879+0000 7fc7df7fe640 10 client.?.objecter ms_handle_connect 0x7fc7f81473a0 2024-03-25T18:15:32.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.879+0000 7fc7df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:32.888 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.888+0000 7fc7df7fe640 10 client.?.objecter ms_handle_connect 0x7fc7f80b1ef0 2024-03-25T18:15:32.888 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.888+0000 7fc7df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:32.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.891+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531282371 0 0) 0x7fc7f8070280 con 0x7fc7f81473a0 2024-03-25T18:15:32.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.891+0000 7fc7df7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:32.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.891+0000 7fc7df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:32.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.891+0000 7fc7df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:32.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.891+0000 7fc7df7fe640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:15:32.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.891+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e4004eb0 con 0x7fc7f81473a0 2024-03-25T18:15:32.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.894+0000 7fc7df7fe640 10 client.?.objecter ms_handle_connect 0x7fc7f8150070 2024-03-25T18:15:32.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.894+0000 7fc7df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:32.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.902+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300080620 0 0) 0x7fc7f81dbaa0 con 0x7fc7f8150070 2024-03-25T18:15:32.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.902+0000 7fc7df7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:32.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.902+0000 7fc7df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:32.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.902+0000 7fc7df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:32.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.902+0000 7fc7df7fe640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:15:32.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.902+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f8070280 con 0x7fc7f8150070 2024-03-25T18:15:32.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.905+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017029607 0 0) 0x7fc7f8146e80 con 0x7fc7f80b1ef0 2024-03-25T18:15:32.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.905+0000 7fc7df7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:32.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.905+0000 7fc7df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:32.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.905+0000 7fc7df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:32.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.905+0000 7fc7df7fe640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:15:32.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.905+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f81dbaa0 con 0x7fc7f80b1ef0 2024-03-25T18:15:32.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.918+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1218757759 0 0) 0x7fc7e4004eb0 con 0x7fc7f81473a0 2024-03-25T18:15:32.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.918+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7f8146e80 con 0x7fc7f81473a0 2024-03-25T18:15:32.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.918+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc7ec001b00 con 0x7fc7f81473a0 2024-03-25T18:15:32.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.918+0000 7fc7df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:32.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.918+0000 7fc7df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:32.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.918+0000 7fc7df7fe640 10 monclient(hunting): dump: 2024-03-25T18:15:32.920 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:32.920 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:32.920 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:32.920 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:32.920 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:32.920 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:32.920 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:32.920 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:32.920 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:32.920 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:32.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.918+0000 7fc7df7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:32.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.934+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3697350558 0 0) 0x7fc7f8070280 con 0x7fc7f8150070 2024-03-25T18:15:32.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.934+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7e4004eb0 con 0x7fc7f8150070 2024-03-25T18:15:32.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.934+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc7f0001c10 con 0x7fc7f8150070 2024-03-25T18:15:32.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.935+0000 7fc7df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:32.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.935+0000 7fc7df7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:32.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.935+0000 7fc7df7fe640 10 monclient(hunting): dump: 2024-03-25T18:15:32.936 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:32.936 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:32.936 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:32.936 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:32.936 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:32.936 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:32.936 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:32.936 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:32.937 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:32.937 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:32.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2774349045 0 0) 0x7fc7f8146e80 con 0x7fc7f81473a0 2024-03-25T18:15:32.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 >> v1:172.21.15.63:6790/0 conn(0x7fc7f80b1ef0 legacy=0x7fc7f81dbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:32.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 >> v1:172.21.15.63:6789/0 conn(0x7fc7f8150070 legacy=0x7fc7f81458e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:32.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7df7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:32.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7df7fe640 1 monclient: found mon.b 2024-03-25T18:15:32.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:32.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7df7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:32.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f8144560 con 0x7fc7f81473a0 2024-03-25T18:15:32.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7df7fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:32.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7df7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:32.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:32.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7fd433640 5 monclient: authenticate success, global_id 4360 2024-03-25T18:15:32.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7fd433640 10 monclient: _renew_subs 2024-03-25T18:15:32.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7fd433640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:32.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7fd433640 1 -- 172.21.15.63:0/3128021875 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f81dc640 con 0x7fc7f81473a0 2024-03-25T18:15:32.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7fd433640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:32.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7fd433640 10 monclient: _renew_subs 2024-03-25T18:15:32.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7fd433640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:32.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7fd433640 1 -- 172.21.15.63:0/3128021875 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f80710d0 con 0x7fc7f81473a0 2024-03-25T18:15:32.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.950+0000 7fc7fd433640 1 librados: init done 2024-03-25T18:15:32.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.951+0000 7fc7fd433640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:32.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.951+0000 7fc7fd433640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:32.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.951+0000 7fc7fd433640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:32.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:32.951+0000 7fc7fd433640 1 -- 172.21.15.63:0/3128021875 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f80627b0 con 0x7fc7f81473a0 2024-03-25T18:15:33.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.243+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7ec004a90 con 0x7fc7f81473a0 2024-03-25T18:15:33.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.243+0000 7fc7df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:33.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.243+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc7ec004da0 con 0x7fc7f81473a0 2024-03-25T18:15:33.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.243+0000 7fc7df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:33.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.243+0000 7fc7df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:33.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.243+0000 7fc7df7fe640 10 monclient: dump: 2024-03-25T18:15:33.245 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:33.245 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:33.245 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:33.245 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:33.245 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:33.245 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:33.245 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:33.245 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:33.245 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:33.246 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:33.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.243+0000 7fc7f57fa640 4 set_mon_vals no callback set 2024-03-25T18:15:33.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.245+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fc7ec01ad70 con 0x7fc7f81473a0 2024-03-25T18:15:33.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.245+0000 7fc7f6ffd640 10 monclient: get_auth_request con 0x7fc7e406df70 auth_method 0 2024-03-25T18:15:33.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.246+0000 7fc7df7fe640 10 client.4360.objecter ms_handle_connect 0x7fc7e406df70 2024-03-25T18:15:33.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.249+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fc7ec054820 con 0x7fc7f81473a0 2024-03-25T18:15:33.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.249+0000 7fc7df7fe640 10 client.4360.objecter ms_dispatch 0x7fc7f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:33.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.249+0000 7fc7df7fe640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:33.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.249+0000 7fc7df7fe640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:33.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.249+0000 7fc7df7fe640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-25T18:15:33.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.250+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc7ec050330 con 0x7fc7f81473a0 2024-03-25T18:15:33.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.250+0000 7fc7df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:33.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.250+0000 7fc7df7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:33.474 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:03:15.166 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:33.482 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:03:15.174 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:33.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.484+0000 7fc7fd433640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:33.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.484+0000 7fc7fd433640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:33.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.484+0000 7fc7fd433640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:33.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.484+0000 7fc7fd433640 1 -- 172.21.15.63:0/3128021875 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7f81446f0 con 0x7fc7f81473a0 2024-03-25T18:15:33.511 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:03:15.203 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:33.530 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:03:15.191 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:33.531 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:03:15.223 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:33.544 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:03:15.237 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:33.551 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:03:15.212 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:33.565 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:03:15.257 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:33.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.567+0000 7fc7df7fe640 1 -- 172.21.15.63:0/3128021875 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fc7ec052260 con 0x7fc7f81473a0 2024-03-25T18:15:33.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.567+0000 7fc7df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:33.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.567+0000 7fc7df7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:33.568 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:15:33.568 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:33.571 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.570+0000 7fc7fd433640 10 librados: watch_flush enter 2024-03-25T18:15:33.571 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.570+0000 7fc7fd433640 10 librados: watch_flush exit 2024-03-25T18:15:33.571 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.570+0000 7fc7fd433640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-25T18:15:33.571 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.570+0000 7fc7fd433640 10 client.4360.objecter successfully canceled tick 2024-03-25T18:15:33.571 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.570+0000 7fc7fd433640 1 -- 172.21.15.63:0/3128021875 >> v1:172.21.15.81:6800/3640340873 conn(0x7fc7e406df70 legacy=0x7fc7e4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:33.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.570+0000 7fc7fd433640 10 monclient: shutdown 2024-03-25T18:15:33.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.570+0000 7fc7fd433640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:33.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.570+0000 7fc7fd433640 1 -- 172.21.15.63:0/3128021875 >> v1:172.21.15.81:6789/0 conn(0x7fc7f81473a0 legacy=0x7fc7f80707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:33.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.570+0000 7fc7fd433640 1 -- 172.21.15.63:0/3128021875 shutdown_connections 2024-03-25T18:15:33.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.570+0000 7fc7fd433640 1 -- 172.21.15.63:0/3128021875 >> 172.21.15.63:0/3128021875 conn(0x7fc7f80b7ea0 msgr2=0x7fc7f81416c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:33.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.570+0000 7fc7fd433640 1 -- 172.21.15.63:0/3128021875 shutdown_connections 2024-03-25T18:15:33.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.570+0000 7fc7fd433640 1 -- 172.21.15.63:0/3128021875 wait complete. 2024-03-25T18:15:33.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:33.570+0000 7fc7fd433640 1 librados: shutdown 2024-03-25T18:15:33.575 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:03:15.236 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:33.583 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:40.390 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:40.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.501+0000 7f7b82f7b640 10 monclient: get_monmap_and_config 2024-03-25T18:15:40.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.501+0000 7f7b82f7b640 10 monclient: build_initial_monmap 2024-03-25T18:15:40.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.501+0000 7f7b82f7b640 10 monclient: monmap: 2024-03-25T18:15:40.502 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:15:40.502 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:40.502 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:15:40.502305+0000 2024-03-25T18:15:40.502 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:15:40.502305+0000 2024-03-25T18:15:40.502 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:40.502 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:40.503 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:15:40.503 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:15:40.503 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:15:40.503 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:40.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.501+0000 7f7b82f7b640 1 Processor -- start 2024-03-25T18:15:40.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.502+0000 7f7b82f7b640 1 -- start start 2024-03-25T18:15:40.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.502+0000 7f7b82f7b640 10 monclient: init 2024-03-25T18:15:40.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.502+0000 7f7b82f7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:40.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.502+0000 7f7b82f7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:40.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.502+0000 7f7b82f7b640 10 monclient(hunting): picked mon.noname-a con 0x7f7b7c0a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:15:40.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.502+0000 7f7b82f7b640 10 monclient(hunting): picked mon.noname-c con 0x7f7b7c13f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:15:40.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.503+0000 7f7b82f7b640 10 monclient(hunting): picked mon.noname-b con 0x7f7b7c1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:15:40.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.503+0000 7f7b82f7b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c142520 con 0x7f7b7c0a98f0 2024-03-25T18:15:40.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.503+0000 7f7b82f7b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c0b28a0 con 0x7f7b7c13f710 2024-03-25T18:15:40.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.503+0000 7f7b82f7b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c0b2a30 con 0x7f7b7c1430c0 2024-03-25T18:15:40.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.503+0000 7f7b82f7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:40.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.503+0000 7f7b82f7b640 10 monclient(hunting): authenticate will time out at 7634.689374s 2024-03-25T18:15:40.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.509+0000 7f7b80d17640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f7b7c0a98f0 0x7f7b7c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:52274/0 (socket says 172.21.15.63:52274) 2024-03-25T18:15:40.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.509+0000 7f7b80d17640 1 -- 172.21.15.63:0/2360566472 learned_addr learned my addr 172.21.15.63:0/2360566472 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:15:40.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.517+0000 7f7b7b7fe640 1 -- 172.21.15.63:0/2360566472 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368757581 0 0) 0x7f7b7c142520 con 0x7f7b7c0a98f0 2024-03-25T18:15:40.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.517+0000 7f7b7b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:40.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.517+0000 7f7b7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:40.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.517+0000 7f7b7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:40.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.517+0000 7f7b7b7fe640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:15:40.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.517+0000 7f7b7b7fe640 1 -- 172.21.15.63:0/2360566472 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b6c003520 con 0x7f7b7c0a98f0 2024-03-25T18:15:40.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b7b7fe640 1 -- 172.21.15.63:0/2360566472 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1163708340 0 0) 0x7f7b6c003520 con 0x7f7b7c0a98f0 2024-03-25T18:15:40.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b7b7fe640 1 -- 172.21.15.63:0/2360566472 >> v1:172.21.15.63:6790/0 conn(0x7f7b7c1430c0 legacy=0x7f7b7c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:15:40.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b7b7fe640 1 -- 172.21.15.63:0/2360566472 >> v1:172.21.15.81:6789/0 conn(0x7f7b7c13f710 legacy=0x7f7b7c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:40.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b7b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:40.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b7b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:15:40.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:40.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b7b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:15:40.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b7b7fe640 1 -- 172.21.15.63:0/2360566472 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7c0b44c0 con 0x7f7b7c0a98f0 2024-03-25T18:15:40.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b7b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:40.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b7b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:40.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:40.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b7b7fe640 1 -- 172.21.15.63:0/2360566472 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7b70001690 con 0x7f7b7c0a98f0 2024-03-25T18:15:40.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b82f7b640 5 monclient: authenticate success, global_id 4347 2024-03-25T18:15:40.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b82f7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:40.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:40.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b7b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:40.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b7b7fe640 10 monclient: dump: 2024-03-25T18:15:40.537 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:40.537 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:40.538 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:40.538 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:40.538 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:40.538 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:40.538 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:40.538 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:40.538 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:40.538 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:40.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.535+0000 7f7b82f7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:40.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.548+0000 7f7b7b7fe640 1 -- 172.21.15.63:0/2360566472 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b70001860 con 0x7f7b7c0a98f0 2024-03-25T18:15:40.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.548+0000 7f7b7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:40.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.548+0000 7f7b7b7fe640 1 -- 172.21.15.63:0/2360566472 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7b70001c70 con 0x7f7b7c0a98f0 2024-03-25T18:15:40.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.548+0000 7f7b82f7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:40.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.548+0000 7f7b82f7b640 4 set_mon_vals no callback set 2024-03-25T18:15:40.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.548+0000 7f7b7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:40.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.548+0000 7f7b7b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:40.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.548+0000 7f7b7b7fe640 10 monclient: dump: 2024-03-25T18:15:40.549 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:40.549 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:40.549 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:40.549 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:40.549 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:40.549 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:40.550 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:40.550 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:40.550 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:40.550 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:40.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.549+0000 7f7b82f7b640 10 monclient: shutdown 2024-03-25T18:15:40.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.549+0000 7f7b82f7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:40.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.549+0000 7f7b82f7b640 1 -- 172.21.15.63:0/2360566472 >> v1:172.21.15.63:6789/0 conn(0x7f7b7c0a98f0 legacy=0x7f7b7c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:40.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.549+0000 7f7b82f7b640 1 -- 172.21.15.63:0/2360566472 shutdown_connections 2024-03-25T18:15:40.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.549+0000 7f7b82f7b640 1 -- 172.21.15.63:0/2360566472 >> 172.21.15.63:0/2360566472 conn(0x7f7b7c0b7ea0 msgr2=0x7f7b7c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:40.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.549+0000 7f7b82f7b640 1 -- 172.21.15.63:0/2360566472 shutdown_connections 2024-03-25T18:15:40.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.549+0000 7f7b82f7b640 1 -- 172.21.15.63:0/2360566472 wait complete. 2024-03-25T18:15:40.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.549+0000 7f7b82f7b640 10 monclient: build_initial_monmap 2024-03-25T18:15:40.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.549+0000 7f7b82f7b640 10 monclient: monmap: 2024-03-25T18:15:40.552 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:15:40.552 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:40.552 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:15:40.552 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:15:40.552 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:40.552 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:15:40.552 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:15:40.552 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:15:40.552 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:15:40.552 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:40.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.550+0000 7f7b82f7b640 1 librados: starting msgr at 2024-03-25T18:15:40.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.550+0000 7f7b82f7b640 1 librados: starting objecter 2024-03-25T18:15:40.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.550+0000 7f7b82f7b640 1 Processor -- start 2024-03-25T18:15:40.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.550+0000 7f7b82f7b640 1 -- start start 2024-03-25T18:15:40.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.550+0000 7f7b82f7b640 1 librados: setting wanted keys 2024-03-25T18:15:40.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.550+0000 7f7b82f7b640 1 librados: calling monclient init 2024-03-25T18:15:40.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.550+0000 7f7b82f7b640 10 monclient: init 2024-03-25T18:15:40.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.551+0000 7f7b82f7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:40.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.551+0000 7f7b82f7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:40.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.551+0000 7f7b82f7b640 10 monclient(hunting): picked mon.noname-b con 0x7f7b7c0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:15:40.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.551+0000 7f7b82f7b640 10 monclient(hunting): picked mon.noname-c con 0x7f7b7c13f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:15:40.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.551+0000 7f7b82f7b640 10 monclient(hunting): picked mon.noname-a con 0x7f7b7c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:15:40.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.551+0000 7f7b82f7b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c0abde0 con 0x7f7b7c1430c0 2024-03-25T18:15:40.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.551+0000 7f7b82f7b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c1d7880 con 0x7f7b7c0a98f0 2024-03-25T18:15:40.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.551+0000 7f7b82f7b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c1d8470 con 0x7f7b7c13f710 2024-03-25T18:15:40.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.551+0000 7f7b82f7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:40.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.551+0000 7f7b82f7b640 10 monclient(hunting): authenticate will time out at 7634.737498s 2024-03-25T18:15:40.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.559+0000 7f7b81518640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f7b7c1430c0 0x7f7b7c138480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:52292/0 (socket says 172.21.15.63:52292) 2024-03-25T18:15:40.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.559+0000 7f7b81518640 1 -- 172.21.15.63:0/2945353105 learned_addr learned my addr 172.21.15.63:0/2945353105 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:15:40.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.560+0000 7f7b797fa640 10 client.?.objecter ms_handle_connect 0x7f7b7c1430c0 2024-03-25T18:15:40.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.560+0000 7f7b797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:40.564 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.564+0000 7f7b797fa640 10 client.?.objecter ms_handle_connect 0x7f7b7c13f710 2024-03-25T18:15:40.564 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.564+0000 7f7b797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:40.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.566+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112452240 0 0) 0x7f7b7c0abde0 con 0x7f7b7c1430c0 2024-03-25T18:15:40.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.566+0000 7f7b797fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:40.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.566+0000 7f7b797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:40.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.566+0000 7f7b797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:40.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.566+0000 7f7b797fa640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:15:40.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.566+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b60004eb0 con 0x7f7b7c1430c0 2024-03-25T18:15:40.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.577+0000 7f7b797fa640 10 client.?.objecter ms_handle_connect 0x7f7b7c0a98f0 2024-03-25T18:15:40.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.577+0000 7f7b797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:40.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.578+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 261935172 0 0) 0x7f7b60004eb0 con 0x7f7b7c1430c0 2024-03-25T18:15:40.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.578+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b7c0abde0 con 0x7f7b7c1430c0 2024-03-25T18:15:40.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.578+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7b74001ad0 con 0x7f7b7c1430c0 2024-03-25T18:15:40.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.578+0000 7f7b797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:40.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.578+0000 7f7b797fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:40.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.578+0000 7f7b797fa640 10 monclient(hunting): dump: 2024-03-25T18:15:40.580 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:40.580 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:40.580 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:40.580 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:40.580 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:40.580 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:40.580 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:40.580 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:40.580 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:40.580 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:40.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.578+0000 7f7b797fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:40.581 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.579+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161409932 0 0) 0x7f7b7c1d8470 con 0x7f7b7c13f710 2024-03-25T18:15:40.581 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.579+0000 7f7b797fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:40.581 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.579+0000 7f7b797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:40.581 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.579+0000 7f7b797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:40.581 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.579+0000 7f7b797fa640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:15:40.581 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.579+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b60004eb0 con 0x7f7b7c13f710 2024-03-25T18:15:40.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.583+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69648970 0 0) 0x7f7b7c1d7880 con 0x7f7b7c0a98f0 2024-03-25T18:15:40.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.583+0000 7f7b797fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:40.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.583+0000 7f7b797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:40.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.583+0000 7f7b797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:40.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.583+0000 7f7b797fa640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:15:40.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.583+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b7c1d8470 con 0x7f7b7c0a98f0 2024-03-25T18:15:40.595 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.593+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2685304517 0 0) 0x7f7b7c0abde0 con 0x7f7b7c1430c0 2024-03-25T18:15:40.595 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.593+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 >> v1:172.21.15.63:6790/0 conn(0x7f7b7c13f710 legacy=0x7f7b7c1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.593+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 >> v1:172.21.15.81:6789/0 conn(0x7f7b7c0a98f0 legacy=0x7f7b7c1d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.593+0000 7f7b797fa640 10 monclient: _finish_hunting 0 2024-03-25T18:15:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.593+0000 7f7b797fa640 1 monclient: found mon.a 2024-03-25T18:15:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.593+0000 7f7b797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.593+0000 7f7b797fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:15:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.593+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7c1d8600 con 0x7f7b7c1430c0 2024-03-25T18:15:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.593+0000 7f7b797fa640 10 monclient: _finish_auth 0 2024-03-25T18:15:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.593+0000 7f7b797fa640 10 monclient: _check_auth_tickets 2024-03-25T18:15:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.593+0000 7f7b797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.593+0000 7f7b82f7b640 5 monclient: authenticate success, global_id 4350 2024-03-25T18:15:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.593+0000 7f7b82f7b640 10 monclient: _renew_subs 2024-03-25T18:15:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.593+0000 7f7b82f7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:15:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.593+0000 7f7b82f7b640 1 -- 172.21.15.63:0/2945353105 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7c1d41e0 con 0x7f7b7c1430c0 2024-03-25T18:15:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.594+0000 7f7b82f7b640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.594+0000 7f7b82f7b640 10 monclient: _renew_subs 2024-03-25T18:15:40.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.594+0000 7f7b82f7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:15:40.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.594+0000 7f7b82f7b640 1 -- 172.21.15.63:0/2945353105 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7c1d5150 con 0x7f7b7c1430c0 2024-03-25T18:15:40.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.594+0000 7f7b82f7b640 1 librados: init done 2024-03-25T18:15:40.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.594+0000 7f7b82f7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:40.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.594+0000 7f7b82f7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:40.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.594+0000 7f7b82f7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:15:40.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.594+0000 7f7b82f7b640 1 -- 172.21.15.63:0/2945353105 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7c1d76e0 con 0x7f7b7c1430c0 2024-03-25T18:15:40.914 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.912+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b74002020 con 0x7f7b7c1430c0 2024-03-25T18:15:40.914 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.913+0000 7f7b797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:40.914 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.913+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7b74004be0 con 0x7f7b7c1430c0 2024-03-25T18:15:40.914 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.913+0000 7f7b797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:40.914 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.913+0000 7f7b797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:40.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.913+0000 7f7b797fa640 10 monclient: dump: 2024-03-25T18:15:40.915 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:40.915 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:40.915 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:40.915 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:40.915 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:40.915 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:40.915 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:40.915 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:40.915 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:40.915 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:40.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.913+0000 7f7b7affd640 4 set_mon_vals no callback set 2024-03-25T18:15:40.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.913+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f7b7401abb0 con 0x7f7b7c1430c0 2024-03-25T18:15:40.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.914+0000 7f7b80d17640 10 monclient: get_auth_request con 0x7f7b6006de10 auth_method 0 2024-03-25T18:15:40.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.914+0000 7f7b797fa640 10 client.4350.objecter ms_handle_connect 0x7f7b6006de10 2024-03-25T18:15:40.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.921+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f7b74054730 con 0x7f7b7c1430c0 2024-03-25T18:15:40.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.921+0000 7f7b797fa640 10 client.4350.objecter ms_dispatch 0x7f7b7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:40.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.921+0000 7f7b797fa640 3 client.4350.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:40.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.921+0000 7f7b797fa640 3 client.4350.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:40.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.921+0000 7f7b797fa640 20 client.4350.objecter dump_active .. 0 homeless 2024-03-25T18:15:40.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.921+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b74050070 con 0x7f7b7c1430c0 2024-03-25T18:15:40.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.921+0000 7f7b797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:40.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:40.921+0000 7f7b797fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:41.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.174+0000 7f7b82f7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:41.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.174+0000 7f7b82f7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:41.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.174+0000 7f7b82f7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:15:41.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.174+0000 7f7b82f7b640 1 -- 172.21.15.63:0/2945353105 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b7c1d8600 con 0x7f7b7c1430c0 2024-03-25T18:15:41.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.202+0000 7f7b797fa640 1 -- 172.21.15.63:0/2945353105 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f7b7c1d8600 con 0x7f7b7c1430c0 2024-03-25T18:15:41.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.202+0000 7f7b797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:41.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.202+0000 7f7b797fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:41.203 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:15:41.203 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:41.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.204+0000 7f7b82f7b640 10 librados: watch_flush enter 2024-03-25T18:15:41.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.204+0000 7f7b82f7b640 10 librados: watch_flush exit 2024-03-25T18:15:41.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.204+0000 7f7b82f7b640 20 client.4350.objecter shutdown clearing up homeless session... 2024-03-25T18:15:41.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.204+0000 7f7b82f7b640 10 client.4350.objecter successfully canceled tick 2024-03-25T18:15:41.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.204+0000 7f7b82f7b640 1 -- 172.21.15.63:0/2945353105 >> v1:172.21.15.81:6800/3640340873 conn(0x7f7b6006de10 legacy=0x7f7b600702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:41.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.204+0000 7f7b82f7b640 10 monclient: shutdown 2024-03-25T18:15:41.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.204+0000 7f7b82f7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:41.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.204+0000 7f7b82f7b640 1 -- 172.21.15.63:0/2945353105 >> v1:172.21.15.63:6789/0 conn(0x7f7b7c1430c0 legacy=0x7f7b7c138480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:41.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.204+0000 7f7b82f7b640 1 -- 172.21.15.63:0/2945353105 shutdown_connections 2024-03-25T18:15:41.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.205+0000 7f7b82f7b640 1 -- 172.21.15.63:0/2945353105 >> 172.21.15.63:0/2945353105 conn(0x7f7b7c0b7ea0 msgr2=0x7f7b7c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:41.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.205+0000 7f7b82f7b640 1 -- 172.21.15.63:0/2945353105 shutdown_connections 2024-03-25T18:15:41.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.205+0000 7f7b82f7b640 1 -- 172.21.15.63:0/2945353105 wait complete. 2024-03-25T18:15:41.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:41.205+0000 7f7b82f7b640 1 librados: shutdown 2024-03-25T18:15:41.217 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:48.023 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:48.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.144+0000 7fe0810a8640 10 monclient: get_monmap_and_config 2024-03-25T18:15:48.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.144+0000 7fe0810a8640 10 monclient: build_initial_monmap 2024-03-25T18:15:48.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.145+0000 7fe0810a8640 10 monclient: monmap: 2024-03-25T18:15:48.146 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:15:48.146 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:48.146 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:15:48.146130+0000 2024-03-25T18:15:48.146 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:15:48.146130+0000 2024-03-25T18:15:48.146 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:48.146 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:48.146 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:15:48.147 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:15:48.147 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:15:48.147 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:48.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.145+0000 7fe0810a8640 1 Processor -- start 2024-03-25T18:15:48.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.145+0000 7fe0810a8640 1 -- start start 2024-03-25T18:15:48.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.145+0000 7fe0810a8640 10 monclient: init 2024-03-25T18:15:48.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.146+0000 7fe0810a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:48.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.146+0000 7fe0810a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:48.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.146+0000 7fe0810a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe07c0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:15:48.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.146+0000 7fe0810a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe07c13f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:15:48.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.146+0000 7fe0810a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe07c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:15:48.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.146+0000 7fe0810a8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07c142520 con 0x7fe07c1430c0 2024-03-25T18:15:48.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.146+0000 7fe0810a8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07c0b28a0 con 0x7fe07c0a98f0 2024-03-25T18:15:48.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.146+0000 7fe0810a8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07c0b2a30 con 0x7fe07c13f710 2024-03-25T18:15:48.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.146+0000 7fe0810a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:48.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.146+0000 7fe0810a8640 10 monclient(hunting): authenticate will time out at 7642.332917s 2024-03-25T18:15:48.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.169+0000 7fe07b59e640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fe07c1430c0 0x7fe07c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:57144/0 (socket says 172.21.15.63:57144) 2024-03-25T18:15:48.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.169+0000 7fe07b59e640 1 -- 172.21.15.63:0/1755932644 learned_addr learned my addr 172.21.15.63:0/1755932644 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:15:48.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.183+0000 7fe079d9b640 1 -- 172.21.15.63:0/1755932644 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518903526 0 0) 0x7fe07c0b28a0 con 0x7fe07c0a98f0 2024-03-25T18:15:48.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.183+0000 7fe079d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:48.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.183+0000 7fe079d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:48.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.183+0000 7fe079d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:48.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.183+0000 7fe079d9b640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:15:48.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.183+0000 7fe079d9b640 1 -- 172.21.15.63:0/1755932644 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe064003520 con 0x7fe07c0a98f0 2024-03-25T18:15:48.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.187+0000 7fe079d9b640 1 -- 172.21.15.63:0/1755932644 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879760480 0 0) 0x7fe07c142520 con 0x7fe07c1430c0 2024-03-25T18:15:48.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.187+0000 7fe079d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:48.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.187+0000 7fe079d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:48.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.187+0000 7fe079d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:48.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.187+0000 7fe079d9b640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:15:48.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.187+0000 7fe079d9b640 1 -- 172.21.15.63:0/1755932644 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe07c0b28a0 con 0x7fe07c1430c0 2024-03-25T18:15:48.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 1 -- 172.21.15.63:0/1755932644 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277260914 0 0) 0x7fe07c0b2a30 con 0x7fe07c13f710 2024-03-25T18:15:48.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:48.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:48.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:48.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:15:48.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 1 -- 172.21.15.63:0/1755932644 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe07c142520 con 0x7fe07c13f710 2024-03-25T18:15:48.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 1 -- 172.21.15.63:0/1755932644 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1314332975 0 0) 0x7fe064003520 con 0x7fe07c0a98f0 2024-03-25T18:15:48.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 1 -- 172.21.15.63:0/1755932644 >> v1:172.21.15.63:6790/0 conn(0x7fe07c13f710 legacy=0x7fe07c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:48.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 1 -- 172.21.15.63:0/1755932644 >> v1:172.21.15.63:6789/0 conn(0x7fe07c1430c0 legacy=0x7fe07c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:48.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:15:48.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 1 monclient: found mon.noname-c 2024-03-25T18:15:48.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:48.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:15:48.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 1 -- 172.21.15.63:0/1755932644 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c0b44c0 con 0x7fe07c0a98f0 2024-03-25T18:15:48.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 10 monclient: _finish_auth 0 2024-03-25T18:15:48.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:15:48.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:48.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 1 -- 172.21.15.63:0/1755932644 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe06c001690 con 0x7fe07c0a98f0 2024-03-25T18:15:48.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe0810a8640 5 monclient: authenticate success, global_id 4369 2024-03-25T18:15:48.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe0810a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:48.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:48.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:48.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe079d9b640 10 monclient: dump: 2024-03-25T18:15:48.198 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:48.198 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:48.198 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:48.198 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:48.198 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:48.198 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:48.198 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:48.198 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:48.198 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:48.198 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:48.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.194+0000 7fe0810a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:48.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.199+0000 7fe079d9b640 1 -- 172.21.15.63:0/1755932644 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe06c0017f0 con 0x7fe07c0a98f0 2024-03-25T18:15:48.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.199+0000 7fe079d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:48.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.199+0000 7fe079d9b640 1 -- 172.21.15.63:0/1755932644 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe06c001c00 con 0x7fe07c0a98f0 2024-03-25T18:15:48.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.199+0000 7fe079d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:48.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.199+0000 7fe079d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:48.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.199+0000 7fe079d9b640 10 monclient: dump: 2024-03-25T18:15:48.200 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:48.200 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:48.200 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:48.200 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:48.200 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:48.201 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:48.201 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:48.201 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:48.201 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:48.201 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:48.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.199+0000 7fe0810a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:48.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.199+0000 7fe0810a8640 4 set_mon_vals no callback set 2024-03-25T18:15:48.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.200+0000 7fe0810a8640 10 monclient: shutdown 2024-03-25T18:15:48.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.200+0000 7fe0810a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:48.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.200+0000 7fe0810a8640 1 -- 172.21.15.63:0/1755932644 >> v1:172.21.15.81:6789/0 conn(0x7fe07c0a98f0 legacy=0x7fe07c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:48.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.200+0000 7fe0810a8640 1 -- 172.21.15.63:0/1755932644 shutdown_connections 2024-03-25T18:15:48.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.200+0000 7fe0810a8640 1 -- 172.21.15.63:0/1755932644 >> 172.21.15.63:0/1755932644 conn(0x7fe07c0b7ea0 msgr2=0x7fe07c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:48.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.200+0000 7fe0810a8640 1 -- 172.21.15.63:0/1755932644 shutdown_connections 2024-03-25T18:15:48.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.200+0000 7fe0810a8640 1 -- 172.21.15.63:0/1755932644 wait complete. 2024-03-25T18:15:48.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.201+0000 7fe0810a8640 10 monclient: build_initial_monmap 2024-03-25T18:15:48.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.201+0000 7fe0810a8640 10 monclient: monmap: 2024-03-25T18:15:48.203 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:15:48.203 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:48.203 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:15:48.203 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:15:48.203 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:48.203 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:15:48.204 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:15:48.204 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:15:48.204 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:15:48.204 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:48.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.201+0000 7fe0810a8640 1 librados: starting msgr at 2024-03-25T18:15:48.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.201+0000 7fe0810a8640 1 librados: starting objecter 2024-03-25T18:15:48.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.202+0000 7fe0810a8640 1 Processor -- start 2024-03-25T18:15:48.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.202+0000 7fe0810a8640 1 -- start start 2024-03-25T18:15:48.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.202+0000 7fe0810a8640 1 librados: setting wanted keys 2024-03-25T18:15:48.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.202+0000 7fe0810a8640 1 librados: calling monclient init 2024-03-25T18:15:48.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.202+0000 7fe0810a8640 10 monclient: init 2024-03-25T18:15:48.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.202+0000 7fe0810a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:48.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.202+0000 7fe0810a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:48.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.202+0000 7fe0810a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe07c0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:15:48.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.202+0000 7fe0810a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe07c13f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:15:48.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.202+0000 7fe0810a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe07c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:15:48.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.202+0000 7fe0810a8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07c135730 con 0x7fe07c1430c0 2024-03-25T18:15:48.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.202+0000 7fe0810a8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07c1358c0 con 0x7fe07c13f710 2024-03-25T18:15:48.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.202+0000 7fe0810a8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07c135a50 con 0x7fe07c0a98f0 2024-03-25T18:15:48.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.202+0000 7fe0810a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:48.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.202+0000 7fe0810a8640 10 monclient(hunting): authenticate will time out at 7642.389216s 2024-03-25T18:15:48.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.218+0000 7fe07a59c640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe07c13f710 0x7fe07c1d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:53168/0 (socket says 172.21.15.63:53168) 2024-03-25T18:15:48.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.218+0000 7fe07a59c640 1 -- 172.21.15.63:0/2354405203 learned_addr learned my addr 172.21.15.63:0/2354405203 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:15:48.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.219+0000 7fe0637fe640 10 client.?.objecter ms_handle_connect 0x7fe07c13f710 2024-03-25T18:15:48.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.219+0000 7fe0637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:48.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.229+0000 7fe0637fe640 10 client.?.objecter ms_handle_connect 0x7fe07c0a98f0 2024-03-25T18:15:48.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.229+0000 7fe0637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:48.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.233+0000 7fe0637fe640 10 client.?.objecter ms_handle_connect 0x7fe07c1430c0 2024-03-25T18:15:48.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.233+0000 7fe0637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:48.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.234+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722022539 0 0) 0x7fe07c1358c0 con 0x7fe07c13f710 2024-03-25T18:15:48.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.234+0000 7fe0637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:48.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.234+0000 7fe0637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:48.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.234+0000 7fe0637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:48.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.234+0000 7fe0637fe640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:15:48.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.234+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe068004eb0 con 0x7fe07c13f710 2024-03-25T18:15:48.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.249+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528987290 0 0) 0x7fe07c135a50 con 0x7fe07c0a98f0 2024-03-25T18:15:48.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.249+0000 7fe0637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:48.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.249+0000 7fe0637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:48.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.249+0000 7fe0637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:48.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.249+0000 7fe0637fe640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:15:48.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.249+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe07c1358c0 con 0x7fe07c0a98f0 2024-03-25T18:15:48.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.250+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1790124929 0 0) 0x7fe068004eb0 con 0x7fe07c13f710 2024-03-25T18:15:48.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.250+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe07c135a50 con 0x7fe07c13f710 2024-03-25T18:15:48.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.250+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe070001a10 con 0x7fe07c13f710 2024-03-25T18:15:48.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.250+0000 7fe0637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:48.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.250+0000 7fe0637fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:48.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.250+0000 7fe0637fe640 10 monclient(hunting): dump: 2024-03-25T18:15:48.251 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:48.251 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:48.251 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:48.251 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:48.251 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:48.251 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:48.252 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:48.252 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:48.252 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:48.252 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:48.252 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.250+0000 7fe0637fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:48.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.254+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010580686 0 0) 0x7fe07c135730 con 0x7fe07c1430c0 2024-03-25T18:15:48.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.254+0000 7fe0637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:48.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.254+0000 7fe0637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:48.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.254+0000 7fe0637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:48.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.254+0000 7fe0637fe640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:15:48.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.254+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe068004eb0 con 0x7fe07c1430c0 2024-03-25T18:15:48.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 77686795 0 0) 0x7fe07c135a50 con 0x7fe07c13f710 2024-03-25T18:15:48.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 >> v1:172.21.15.63:6790/0 conn(0x7fe07c0a98f0 legacy=0x7fe07c1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:48.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 >> v1:172.21.15.63:6789/0 conn(0x7fe07c1430c0 legacy=0x7fe07c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:48.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0637fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:48.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0637fe640 1 monclient: found mon.b 2024-03-25T18:15:48.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:48.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0637fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:48.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c1d5000 con 0x7fe07c13f710 2024-03-25T18:15:48.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0637fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:48.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0637fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:48.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:48.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0810a8640 5 monclient: authenticate success, global_id 4372 2024-03-25T18:15:48.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0810a8640 10 monclient: _renew_subs 2024-03-25T18:15:48.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0810a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:48.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0810a8640 1 -- 172.21.15.63:0/2354405203 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07c1d52b0 con 0x7fe07c13f710 2024-03-25T18:15:48.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0810a8640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:48.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0810a8640 10 monclient: _renew_subs 2024-03-25T18:15:48.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0810a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:48.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0810a8640 1 -- 172.21.15.63:0/2354405203 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07c13c900 con 0x7fe07c13f710 2024-03-25T18:15:48.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.260+0000 7fe0810a8640 1 librados: init done 2024-03-25T18:15:48.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.261+0000 7fe0810a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:48.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.261+0000 7fe0810a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:48.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.261+0000 7fe0810a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:48.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.261+0000 7fe0810a8640 1 -- 172.21.15.63:0/2354405203 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe07c1371c0 con 0x7fe07c13f710 2024-03-25T18:15:48.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.534+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe070001c70 con 0x7fe07c13f710 2024-03-25T18:15:48.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.534+0000 7fe0637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:48.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.534+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe070003a60 con 0x7fe07c13f710 2024-03-25T18:15:48.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.534+0000 7fe0637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:48.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.534+0000 7fe0637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:48.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.534+0000 7fe0637fe640 10 monclient: dump: 2024-03-25T18:15:48.536 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:48.536 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:48.536 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:48.536 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:48.536 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:48.536 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:48.536 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:48.536 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:48.536 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:48.536 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:48.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.534+0000 7fe07959a640 4 set_mon_vals no callback set 2024-03-25T18:15:48.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.535+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fe070003c80 con 0x7fe07c13f710 2024-03-25T18:15:48.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.536+0000 7fe07ad9d640 10 monclient: get_auth_request con 0x7fe06806de00 auth_method 0 2024-03-25T18:15:48.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.536+0000 7fe0637fe640 10 client.4372.objecter ms_handle_connect 0x7fe06806de00 2024-03-25T18:15:48.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.539+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fe070054820 con 0x7fe07c13f710 2024-03-25T18:15:48.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.539+0000 7fe0637fe640 10 client.4372.objecter ms_dispatch 0x7fe07c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:48.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.539+0000 7fe0637fe640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:48.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.539+0000 7fe0637fe640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:48.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.539+0000 7fe0637fe640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-25T18:15:48.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.539+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe070050330 con 0x7fe07c13f710 2024-03-25T18:15:48.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.539+0000 7fe0637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:48.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.539+0000 7fe0637fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:48.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.793+0000 7fe0810a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:48.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.793+0000 7fe0810a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:48.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.793+0000 7fe0810a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:15:48.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.793+0000 7fe0810a8640 1 -- 172.21.15.63:0/2354405203 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe07c0627b0 con 0x7fe07c13f710 2024-03-25T18:15:48.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.827+0000 7fe0637fe640 1 -- 172.21.15.63:0/2354405203 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fe070052260 con 0x7fe07c13f710 2024-03-25T18:15:48.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.827+0000 7fe0637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:48.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.827+0000 7fe0637fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:48.828 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:15:48.828 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:48.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.829+0000 7fe0810a8640 10 librados: watch_flush enter 2024-03-25T18:15:48.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.829+0000 7fe0810a8640 10 librados: watch_flush exit 2024-03-25T18:15:48.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.829+0000 7fe0810a8640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-25T18:15:48.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.829+0000 7fe0810a8640 10 client.4372.objecter successfully canceled tick 2024-03-25T18:15:48.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.829+0000 7fe0810a8640 1 -- 172.21.15.63:0/2354405203 >> v1:172.21.15.81:6800/3640340873 conn(0x7fe06806de00 legacy=0x7fe0680702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:48.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.829+0000 7fe0810a8640 10 monclient: shutdown 2024-03-25T18:15:48.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.829+0000 7fe0810a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:48.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.829+0000 7fe0810a8640 1 -- 172.21.15.63:0/2354405203 >> v1:172.21.15.81:6789/0 conn(0x7fe07c13f710 legacy=0x7fe07c1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:48.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.829+0000 7fe0810a8640 1 -- 172.21.15.63:0/2354405203 shutdown_connections 2024-03-25T18:15:48.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.829+0000 7fe0810a8640 1 -- 172.21.15.63:0/2354405203 >> 172.21.15.63:0/2354405203 conn(0x7fe07c0b7ea0 msgr2=0x7fe07c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:48.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.829+0000 7fe0810a8640 1 -- 172.21.15.63:0/2354405203 shutdown_connections 2024-03-25T18:15:48.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.830+0000 7fe0810a8640 1 -- 172.21.15.63:0/2354405203 wait complete. 2024-03-25T18:15:48.832 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:48.830+0000 7fe0810a8640 1 librados: shutdown 2024-03-25T18:15:48.842 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:53.532 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:03:35.224 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:53.538 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:03:35.231 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:53.568 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:03:35.260 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:53.582 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:03:35.243 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:53.589 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:03:35.281 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:53.596 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:03:35.256 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:53.603 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:03:35.296 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:53.640 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:03:35.300 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:53.641 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:03:35.333 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:55.649 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:55.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.758+0000 7f42c2515640 10 monclient: get_monmap_and_config 2024-03-25T18:15:55.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.758+0000 7f42c2515640 10 monclient: build_initial_monmap 2024-03-25T18:15:55.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.758+0000 7f42c2515640 10 monclient: monmap: 2024-03-25T18:15:55.760 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:15:55.760 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:55.760 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:15:55.759670+0000 2024-03-25T18:15:55.760 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:15:55.759670+0000 2024-03-25T18:15:55.760 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:55.760 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:55.760 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:15:55.760 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:15:55.760 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:15:55.760 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:55.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.759+0000 7f42c2515640 1 Processor -- start 2024-03-25T18:15:55.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.759+0000 7f42c2515640 1 -- start start 2024-03-25T18:15:55.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.759+0000 7f42c2515640 10 monclient: init 2024-03-25T18:15:55.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.759+0000 7f42c2515640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:55.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.759+0000 7f42c2515640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:55.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.760+0000 7f42c2515640 10 monclient(hunting): picked mon.noname-b con 0x7f42bc0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:15:55.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.760+0000 7f42c2515640 10 monclient(hunting): picked mon.noname-c con 0x7f42bc13f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:15:55.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.760+0000 7f42c2515640 10 monclient(hunting): picked mon.noname-a con 0x7f42bc1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:15:55.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.760+0000 7f42c2515640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc142520 con 0x7f42bc1430c0 2024-03-25T18:15:55.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.760+0000 7f42c2515640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc0b28a0 con 0x7f42bc13f710 2024-03-25T18:15:55.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.760+0000 7f42c2515640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc0b2a30 con 0x7f42bc0a98f0 2024-03-25T18:15:55.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.760+0000 7f42c2515640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:55.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.760+0000 7f42c2515640 10 monclient(hunting): authenticate will time out at 7649.946376s 2024-03-25T18:15:55.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.772+0000 7f42c0ab2640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f42bc1430c0 0x7f42bc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:45934/0 (socket says 172.21.15.63:45934) 2024-03-25T18:15:55.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.772+0000 7f42c0ab2640 1 -- 172.21.15.63:0/2494411824 learned_addr learned my addr 172.21.15.63:0/2494411824 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:15:55.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.790+0000 7f42baffd640 1 -- 172.21.15.63:0/2494411824 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093517805 0 0) 0x7f42bc0b2a30 con 0x7f42bc0a98f0 2024-03-25T18:15:55.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.790+0000 7f42baffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:55.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.790+0000 7f42baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:55.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.790+0000 7f42baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:55.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.790+0000 7f42baffd640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:15:55.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.790+0000 7f42baffd640 1 -- 172.21.15.63:0/2494411824 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42a4003520 con 0x7f42bc0a98f0 2024-03-25T18:15:55.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.794+0000 7f42baffd640 1 -- 172.21.15.63:0/2494411824 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832909473 0 0) 0x7f42bc142520 con 0x7f42bc1430c0 2024-03-25T18:15:55.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.794+0000 7f42baffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:55.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.794+0000 7f42baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:55.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.794+0000 7f42baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:55.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.794+0000 7f42baffd640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:15:55.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.794+0000 7f42baffd640 1 -- 172.21.15.63:0/2494411824 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42bc0b2a30 con 0x7f42bc1430c0 2024-03-25T18:15:55.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42baffd640 1 -- 172.21.15.63:0/2494411824 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 616818123 0 0) 0x7f42a4003520 con 0x7f42bc0a98f0 2024-03-25T18:15:55.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42baffd640 1 -- 172.21.15.63:0/2494411824 >> v1:172.21.15.81:6789/0 conn(0x7f42bc13f710 legacy=0x7f42bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:55.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42baffd640 1 -- 172.21.15.63:0/2494411824 >> v1:172.21.15.63:6789/0 conn(0x7f42bc1430c0 legacy=0x7f42bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:55.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42baffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:55.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42baffd640 1 monclient: found mon.noname-b 2024-03-25T18:15:55.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:55.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42baffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:15:55.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42baffd640 1 -- 172.21.15.63:0/2494411824 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc0b44c0 con 0x7f42bc0a98f0 2024-03-25T18:15:55.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42baffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:55.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42baffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:55.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:55.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42baffd640 1 -- 172.21.15.63:0/2494411824 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f42ac001690 con 0x7f42bc0a98f0 2024-03-25T18:15:55.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:55.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42baffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:55.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42baffd640 10 monclient: dump: 2024-03-25T18:15:55.803 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:55.803 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:55.803 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:55.803 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:55.803 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:55.803 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:55.803 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:55.803 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:55.803 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:55.803 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:55.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42c2515640 5 monclient: authenticate success, global_id 4361 2024-03-25T18:15:55.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.800+0000 7f42c2515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:55.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.808+0000 7f42baffd640 1 -- 172.21.15.63:0/2494411824 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42ac0017f0 con 0x7f42bc0a98f0 2024-03-25T18:15:55.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.808+0000 7f42baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:55.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.809+0000 7f42baffd640 1 -- 172.21.15.63:0/2494411824 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f42ac001c00 con 0x7f42bc0a98f0 2024-03-25T18:15:55.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.809+0000 7f42baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:55.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.809+0000 7f42baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:55.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.809+0000 7f42baffd640 10 monclient: dump: 2024-03-25T18:15:55.810 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:55.810 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:55.810 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:55.810 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:55.810 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:55.810 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:55.811 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:55.811 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:55.811 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:55.811 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:55.811 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.809+0000 7f42c2515640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:55.811 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.809+0000 7f42c2515640 4 set_mon_vals no callback set 2024-03-25T18:15:55.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.809+0000 7f42c2515640 10 monclient: shutdown 2024-03-25T18:15:55.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.809+0000 7f42c2515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:55.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.809+0000 7f42c2515640 1 -- 172.21.15.63:0/2494411824 >> v1:172.21.15.63:6790/0 conn(0x7f42bc0a98f0 legacy=0x7f42bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:55.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.810+0000 7f42c2515640 1 -- 172.21.15.63:0/2494411824 shutdown_connections 2024-03-25T18:15:55.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.810+0000 7f42c2515640 1 -- 172.21.15.63:0/2494411824 >> 172.21.15.63:0/2494411824 conn(0x7f42bc0b7ea0 msgr2=0x7f42bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:55.813 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.810+0000 7f42c2515640 1 -- 172.21.15.63:0/2494411824 shutdown_connections 2024-03-25T18:15:55.814 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.810+0000 7f42c2515640 1 -- 172.21.15.63:0/2494411824 wait complete. 2024-03-25T18:15:55.814 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.811+0000 7f42c2515640 10 monclient: build_initial_monmap 2024-03-25T18:15:55.814 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.811+0000 7f42c2515640 10 monclient: monmap: 2024-03-25T18:15:55.814 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:15:55.814 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:55.814 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:15:55.814 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:15:55.814 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:55.814 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:15:55.814 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:15:55.814 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:15:55.814 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:15:55.815 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:55.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.811+0000 7f42c2515640 1 librados: starting msgr at 2024-03-25T18:15:55.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.811+0000 7f42c2515640 1 librados: starting objecter 2024-03-25T18:15:55.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.811+0000 7f42c2515640 1 Processor -- start 2024-03-25T18:15:55.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.811+0000 7f42c2515640 1 -- start start 2024-03-25T18:15:55.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.811+0000 7f42c2515640 1 librados: setting wanted keys 2024-03-25T18:15:55.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.811+0000 7f42c2515640 1 librados: calling monclient init 2024-03-25T18:15:55.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.811+0000 7f42c2515640 10 monclient: init 2024-03-25T18:15:55.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.812+0000 7f42c2515640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:55.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.812+0000 7f42c2515640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:55.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.812+0000 7f42c2515640 10 monclient(hunting): picked mon.noname-b con 0x7f42bc0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:15:55.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.812+0000 7f42c2515640 10 monclient(hunting): picked mon.noname-a con 0x7f42bc13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:15:55.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.812+0000 7f42c2515640 10 monclient(hunting): picked mon.noname-c con 0x7f42bc1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:15:55.816 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.812+0000 7f42c2515640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc1370a0 con 0x7f42bc13f710 2024-03-25T18:15:55.816 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.812+0000 7f42c2515640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc135090 con 0x7f42bc0a98f0 2024-03-25T18:15:55.816 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.812+0000 7f42c2515640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc135220 con 0x7f42bc1430c0 2024-03-25T18:15:55.816 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.812+0000 7f42c2515640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:55.816 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.812+0000 7f42c2515640 10 monclient(hunting): authenticate will time out at 7649.998736s 2024-03-25T18:15:55.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.829+0000 7f42bbfff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f42bc0a98f0 0x7f42bc1dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:53668/0 (socket says 172.21.15.63:53668) 2024-03-25T18:15:55.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.829+0000 7f42bbfff640 1 -- 172.21.15.63:0/524257983 learned_addr learned my addr 172.21.15.63:0/524257983 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:15:55.834 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.834+0000 7f42b8ff9640 10 client.?.objecter ms_handle_connect 0x7f42bc13f710 2024-03-25T18:15:55.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.834+0000 7f42b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:55.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.835+0000 7f42b8ff9640 10 client.?.objecter ms_handle_connect 0x7f42bc0a98f0 2024-03-25T18:15:55.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.835+0000 7f42b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:55.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.838+0000 7f42b8ff9640 10 client.?.objecter ms_handle_connect 0x7f42bc1430c0 2024-03-25T18:15:55.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.838+0000 7f42b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:55.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.838+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807153179 0 0) 0x7f42bc1370a0 con 0x7f42bc13f710 2024-03-25T18:15:55.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.838+0000 7f42b8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:55.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.838+0000 7f42b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:55.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.838+0000 7f42b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:55.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.838+0000 7f42b8ff9640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:15:55.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.839+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42a8004eb0 con 0x7f42bc13f710 2024-03-25T18:15:55.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.848+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050465665 0 0) 0x7f42bc135090 con 0x7f42bc0a98f0 2024-03-25T18:15:55.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.848+0000 7f42b8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:55.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.848+0000 7f42b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:55.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.848+0000 7f42b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:55.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.848+0000 7f42b8ff9640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:15:55.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.848+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42bc1370a0 con 0x7f42bc0a98f0 2024-03-25T18:15:55.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.858+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001015449 0 0) 0x7f42bc135220 con 0x7f42bc1430c0 2024-03-25T18:15:55.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.858+0000 7f42b8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:55.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.858+0000 7f42b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:55.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.858+0000 7f42b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:55.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.858+0000 7f42b8ff9640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:15:55.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.858+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42bc135090 con 0x7f42bc1430c0 2024-03-25T18:15:55.860 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.859+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1508096526 0 0) 0x7f42a8004eb0 con 0x7f42bc13f710 2024-03-25T18:15:55.860 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.859+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42bc135220 con 0x7f42bc13f710 2024-03-25T18:15:55.860 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.859+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f42b0001930 con 0x7f42bc13f710 2024-03-25T18:15:55.860 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.859+0000 7f42b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:55.860 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.859+0000 7f42b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:55.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.859+0000 7f42b8ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:55.861 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:55.861 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:55.861 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:55.861 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:55.861 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:55.861 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:55.861 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:55.861 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:55.861 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:55.861 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:55.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.859+0000 7f42b8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:55.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1861331256 0 0) 0x7f42bc135220 con 0x7f42bc13f710 2024-03-25T18:15:55.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 >> v1:172.21.15.63:6790/0 conn(0x7f42bc1430c0 legacy=0x7f42bc1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:55.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 >> v1:172.21.15.81:6789/0 conn(0x7f42bc0a98f0 legacy=0x7f42bc1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:55.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42b8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:15:55.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42b8ff9640 1 monclient: found mon.a 2024-03-25T18:15:55.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:55.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42b8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:15:55.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc1d5000 con 0x7f42bc13f710 2024-03-25T18:15:55.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42b8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:15:55.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42b8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:15:55.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:55.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42c2515640 5 monclient: authenticate success, global_id 4362 2024-03-25T18:15:55.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42c2515640 10 monclient: _renew_subs 2024-03-25T18:15:55.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42c2515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:15:55.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42c2515640 1 -- 172.21.15.63:0/524257983 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42bc1d52b0 con 0x7f42bc13f710 2024-03-25T18:15:55.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42c2515640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:55.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42c2515640 10 monclient: _renew_subs 2024-03-25T18:15:55.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42c2515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:15:55.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42c2515640 1 -- 172.21.15.63:0/524257983 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42bc135910 con 0x7f42bc13f710 2024-03-25T18:15:55.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.876+0000 7f42c2515640 1 librados: init done 2024-03-25T18:15:55.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.877+0000 7f42c2515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:55.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.877+0000 7f42c2515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:55.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.877+0000 7f42c2515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:15:55.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:55.877+0000 7f42c2515640 1 -- 172.21.15.63:0/524257983 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42bc0627b0 con 0x7f42bc13f710 2024-03-25T18:15:56.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.193+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42b0003610 con 0x7f42bc13f710 2024-03-25T18:15:56.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.193+0000 7f42b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:56.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.193+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f42b0003a20 con 0x7f42bc13f710 2024-03-25T18:15:56.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.193+0000 7f42b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:56.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.193+0000 7f42b8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:56.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.193+0000 7f42b8ff9640 10 monclient: dump: 2024-03-25T18:15:56.253 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:15:56.253 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:15:56.253 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:56.254 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:15:56.254 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:56.254 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:15:56.254 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:15:56.254 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:15:56.254 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:15:56.254 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:15:56.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.193+0000 7f42ba7fc640 4 set_mon_vals no callback set 2024-03-25T18:15:56.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.194+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f42b0003c40 con 0x7f42bc13f710 2024-03-25T18:15:56.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.194+0000 7f42bbfff640 10 monclient: get_auth_request con 0x7f42a806ddf0 auth_method 0 2024-03-25T18:15:56.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.194+0000 7f42b8ff9640 10 client.4362.objecter ms_handle_connect 0x7f42a806ddf0 2024-03-25T18:15:56.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.215+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f42b0053660 con 0x7f42bc13f710 2024-03-25T18:15:56.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.215+0000 7f42b8ff9640 10 client.4362.objecter ms_dispatch 0x7f42bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:56.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.215+0000 7f42b8ff9640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:56.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.215+0000 7f42b8ff9640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:56.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.215+0000 7f42b8ff9640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-25T18:15:56.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.215+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f42b004f170 con 0x7f42bc13f710 2024-03-25T18:15:56.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.215+0000 7f42b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:56.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.215+0000 7f42b8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:56.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.469+0000 7f42c2515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:56.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.469+0000 7f42c2515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:56.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.469+0000 7f42c2515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:15:56.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.469+0000 7f42c2515640 1 -- 172.21.15.63:0/524257983 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42bc0008d0 con 0x7f42bc13f710 2024-03-25T18:15:56.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.508+0000 7f42b8ff9640 1 -- 172.21.15.63:0/524257983 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f42b00510a0 con 0x7f42bc13f710 2024-03-25T18:15:56.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.508+0000 7f42b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:56.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.509+0000 7f42b8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:56.510 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:15:56.510 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:56.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.511+0000 7f42c2515640 10 librados: watch_flush enter 2024-03-25T18:15:56.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.511+0000 7f42c2515640 10 librados: watch_flush exit 2024-03-25T18:15:56.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.511+0000 7f42c2515640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-25T18:15:56.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.511+0000 7f42c2515640 10 client.4362.objecter successfully canceled tick 2024-03-25T18:15:56.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.511+0000 7f42c2515640 1 -- 172.21.15.63:0/524257983 >> v1:172.21.15.81:6800/3640340873 conn(0x7f42a806ddf0 legacy=0x7f42a80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:56.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.511+0000 7f42c2515640 10 monclient: shutdown 2024-03-25T18:15:56.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.511+0000 7f42c2515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:56.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.511+0000 7f42c2515640 1 -- 172.21.15.63:0/524257983 >> v1:172.21.15.63:6789/0 conn(0x7f42bc13f710 legacy=0x7f42bc1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:56.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.511+0000 7f42c2515640 1 -- 172.21.15.63:0/524257983 shutdown_connections 2024-03-25T18:15:56.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.511+0000 7f42c2515640 1 -- 172.21.15.63:0/524257983 >> 172.21.15.63:0/524257983 conn(0x7f42bc0b7ea0 msgr2=0x7f42bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:56.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.511+0000 7f42c2515640 1 -- 172.21.15.63:0/524257983 shutdown_connections 2024-03-25T18:15:56.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.511+0000 7f42c2515640 1 -- 172.21.15.63:0/524257983 wait complete. 2024-03-25T18:15:56.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:15:56.511+0000 7f42c2515640 1 librados: shutdown 2024-03-25T18:15:56.524 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:03.332 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:03.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.426+0000 7f619ae25640 10 monclient: get_monmap_and_config 2024-03-25T18:16:03.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.427+0000 7f619ae25640 10 monclient: build_initial_monmap 2024-03-25T18:16:03.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.427+0000 7f619ae25640 10 monclient: monmap: 2024-03-25T18:16:03.428 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:16:03.428 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:03.428 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:16:03.428131+0000 2024-03-25T18:16:03.428 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:16:03.428131+0000 2024-03-25T18:16:03.428 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:03.428 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:03.428 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:16:03.428 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:16:03.428 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:16:03.429 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:03.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.427+0000 7f619ae25640 1 Processor -- start 2024-03-25T18:16:03.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.427+0000 7f619ae25640 1 -- start start 2024-03-25T18:16:03.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.427+0000 7f619ae25640 10 monclient: init 2024-03-25T18:16:03.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.428+0000 7f619ae25640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:03.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.428+0000 7f619ae25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:03.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.428+0000 7f619ae25640 10 monclient(hunting): picked mon.noname-a con 0x7f61940a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:16:03.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.428+0000 7f619ae25640 10 monclient(hunting): picked mon.noname-c con 0x7f619413f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:16:03.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.428+0000 7f619ae25640 10 monclient(hunting): picked mon.noname-b con 0x7f61941430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:16:03.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.428+0000 7f619ae25640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6194142520 con 0x7f61940a98f0 2024-03-25T18:16:03.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.428+0000 7f619ae25640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61940b28a0 con 0x7f619413f710 2024-03-25T18:16:03.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.428+0000 7f619ae25640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61940b2a30 con 0x7f61941430c0 2024-03-25T18:16:03.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.428+0000 7f619ae25640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:03.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.428+0000 7f619ae25640 10 monclient(hunting): authenticate will time out at 7657.615174s 2024-03-25T18:16:03.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.439+0000 7f6198bc1640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f61940a98f0 0x7f61940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:50614/0 (socket says 172.21.15.63:50614) 2024-03-25T18:16:03.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.440+0000 7f6198bc1640 1 -- 172.21.15.63:0/50299539 learned_addr learned my addr 172.21.15.63:0/50299539 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:16:03.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.457+0000 7f618b7fe640 1 -- 172.21.15.63:0/50299539 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222088681 0 0) 0x7f6194142520 con 0x7f61940a98f0 2024-03-25T18:16:03.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.457+0000 7f618b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:03.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.457+0000 7f618b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:03.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.457+0000 7f618b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:03.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.457+0000 7f618b7fe640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:16:03.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.457+0000 7f618b7fe640 1 -- 172.21.15.63:0/50299539 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f617c003520 con 0x7f61940a98f0 2024-03-25T18:16:03.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.462+0000 7f618b7fe640 1 -- 172.21.15.63:0/50299539 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353608537 0 0) 0x7f61940b28a0 con 0x7f619413f710 2024-03-25T18:16:03.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.462+0000 7f618b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:03.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.462+0000 7f618b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:03.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.462+0000 7f618b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:03.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.462+0000 7f618b7fe640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:16:03.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.462+0000 7f618b7fe640 1 -- 172.21.15.63:0/50299539 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6194142520 con 0x7f619413f710 2024-03-25T18:16:03.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f618b7fe640 1 -- 172.21.15.63:0/50299539 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2722623451 0 0) 0x7f6194142520 con 0x7f619413f710 2024-03-25T18:16:03.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f618b7fe640 1 -- 172.21.15.63:0/50299539 >> v1:172.21.15.63:6790/0 conn(0x7f61941430c0 legacy=0x7f619414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:03.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f618b7fe640 1 -- 172.21.15.63:0/50299539 >> v1:172.21.15.63:6789/0 conn(0x7f61940a98f0 legacy=0x7f61940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:03.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f618b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:03.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f618b7fe640 1 monclient: found mon.noname-c 2024-03-25T18:16:03.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f618b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:03.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f618b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:16:03.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f618b7fe640 1 -- 172.21.15.63:0/50299539 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61940b44c0 con 0x7f619413f710 2024-03-25T18:16:03.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f618b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:03.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f618b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:03.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f618b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:03.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f618b7fe640 1 -- 172.21.15.63:0/50299539 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6184001670 con 0x7f619413f710 2024-03-25T18:16:03.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f618b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:03.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f618b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:03.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f618b7fe640 10 monclient: dump: 2024-03-25T18:16:03.470 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:03.470 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:03.470 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:03.470 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:03.470 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:03.470 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:03.470 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:03.470 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:03.470 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:03.470 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:03.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f619ae25640 5 monclient: authenticate success, global_id 4381 2024-03-25T18:16:03.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.467+0000 7f619ae25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:03.480 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.478+0000 7f618b7fe640 1 -- 172.21.15.63:0/50299539 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61840017d0 con 0x7f619413f710 2024-03-25T18:16:03.480 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.478+0000 7f618b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:03.480 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.478+0000 7f618b7fe640 1 -- 172.21.15.63:0/50299539 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6184001ba0 con 0x7f619413f710 2024-03-25T18:16:03.480 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.478+0000 7f619ae25640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:03.480 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.478+0000 7f619ae25640 4 set_mon_vals no callback set 2024-03-25T18:16:03.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.478+0000 7f618b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:03.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.478+0000 7f618b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:03.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.478+0000 7f618b7fe640 10 monclient: dump: 2024-03-25T18:16:03.481 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:03.481 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:03.481 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:03.481 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:03.481 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:03.481 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:03.481 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:03.481 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:03.481 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:03.481 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:03.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.479+0000 7f619ae25640 10 monclient: shutdown 2024-03-25T18:16:03.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.479+0000 7f619ae25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:03.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.479+0000 7f619ae25640 1 -- 172.21.15.63:0/50299539 >> v1:172.21.15.81:6789/0 conn(0x7f619413f710 legacy=0x7f61941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:03.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.479+0000 7f619ae25640 1 -- 172.21.15.63:0/50299539 shutdown_connections 2024-03-25T18:16:03.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.479+0000 7f619ae25640 1 -- 172.21.15.63:0/50299539 >> 172.21.15.63:0/50299539 conn(0x7f61940b7ea0 msgr2=0x7f61940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:03.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.479+0000 7f619ae25640 1 -- 172.21.15.63:0/50299539 shutdown_connections 2024-03-25T18:16:03.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.479+0000 7f619ae25640 1 -- 172.21.15.63:0/50299539 wait complete. 2024-03-25T18:16:03.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.480+0000 7f619ae25640 10 monclient: build_initial_monmap 2024-03-25T18:16:03.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.480+0000 7f619ae25640 10 monclient: monmap: 2024-03-25T18:16:03.483 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:16:03.483 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:03.484 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:16:03.484 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:16:03.484 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:03.484 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:16:03.484 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:16:03.484 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:16:03.484 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:16:03.484 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:03.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.480+0000 7f619ae25640 1 librados: starting msgr at 2024-03-25T18:16:03.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.480+0000 7f619ae25640 1 librados: starting objecter 2024-03-25T18:16:03.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.481+0000 7f619ae25640 1 Processor -- start 2024-03-25T18:16:03.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.481+0000 7f619ae25640 1 -- start start 2024-03-25T18:16:03.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.481+0000 7f619ae25640 1 librados: setting wanted keys 2024-03-25T18:16:03.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.481+0000 7f619ae25640 1 librados: calling monclient init 2024-03-25T18:16:03.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.481+0000 7f619ae25640 10 monclient: init 2024-03-25T18:16:03.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.481+0000 7f619ae25640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:03.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.481+0000 7f619ae25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:03.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.481+0000 7f619ae25640 10 monclient(hunting): picked mon.noname-c con 0x7f61940a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:16:03.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.481+0000 7f619ae25640 10 monclient(hunting): picked mon.noname-b con 0x7f619413f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:16:03.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.481+0000 7f619ae25640 10 monclient(hunting): picked mon.noname-a con 0x7f61941430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:16:03.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.481+0000 7f619ae25640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61941db660 con 0x7f61941430c0 2024-03-25T18:16:03.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.481+0000 7f619ae25640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61941db7f0 con 0x7f619413f710 2024-03-25T18:16:03.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.481+0000 7f619ae25640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619414dee0 con 0x7f61940a98f0 2024-03-25T18:16:03.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.481+0000 7f619ae25640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:03.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.481+0000 7f619ae25640 10 monclient(hunting): authenticate will time out at 7657.668250s 2024-03-25T18:16:03.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.487+0000 7f618bfff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f619413f710 0x7f61940ae760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:41882/0 (socket says 172.21.15.63:41882) 2024-03-25T18:16:03.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.487+0000 7f618bfff640 1 -- 172.21.15.63:0/1228164680 learned_addr learned my addr 172.21.15.63:0/1228164680 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:16:03.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.488+0000 7f61897fa640 10 client.?.objecter ms_handle_connect 0x7f619413f710 2024-03-25T18:16:03.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.488+0000 7f61897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:03.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.493+0000 7f61897fa640 10 client.?.objecter ms_handle_connect 0x7f61940a98f0 2024-03-25T18:16:03.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.493+0000 7f61897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:03.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.493+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304703972 0 0) 0x7f61941db7f0 con 0x7f619413f710 2024-03-25T18:16:03.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.493+0000 7f61897fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:03.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.493+0000 7f61897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:03.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.493+0000 7f61897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:03.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.493+0000 7f61897fa640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:16:03.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.493+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6180004eb0 con 0x7f619413f710 2024-03-25T18:16:03.498 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.497+0000 7f61897fa640 10 client.?.objecter ms_handle_connect 0x7f61941430c0 2024-03-25T18:16:03.498 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.497+0000 7f61897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:03.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.505+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4237015258 0 0) 0x7f6180004eb0 con 0x7f619413f710 2024-03-25T18:16:03.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.505+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61941db7f0 con 0x7f619413f710 2024-03-25T18:16:03.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.505+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f61840035f0 con 0x7f619413f710 2024-03-25T18:16:03.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.505+0000 7f61897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:03.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.505+0000 7f61897fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:03.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.505+0000 7f61897fa640 10 monclient(hunting): dump: 2024-03-25T18:16:03.506 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:03.506 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:03.506 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:03.506 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:03.507 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:03.507 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:03.507 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:03.507 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:03.507 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:03.507 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:03.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.505+0000 7f61897fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:03.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.512+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109401099 0 0) 0x7f619414dee0 con 0x7f61940a98f0 2024-03-25T18:16:03.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.512+0000 7f61897fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:03.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.512+0000 7f61897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:03.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.512+0000 7f61897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:03.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.512+0000 7f61897fa640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:16:03.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.512+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6180004eb0 con 0x7f61940a98f0 2024-03-25T18:16:03.520 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.520+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471272482 0 0) 0x7f61941db660 con 0x7f61941430c0 2024-03-25T18:16:03.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.520+0000 7f61897fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:03.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.520+0000 7f61897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:03.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.520+0000 7f61897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:03.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.520+0000 7f61897fa640 10 monclient(hunting): my global_id is 4368 2024-03-25T18:16:03.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.520+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f619414dee0 con 0x7f61941430c0 2024-03-25T18:16:03.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3348955847 0 0) 0x7f61941db7f0 con 0x7f619413f710 2024-03-25T18:16:03.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 >> v1:172.21.15.63:6790/0 conn(0x7f61940a98f0 legacy=0x7f61941dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:03.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 >> v1:172.21.15.63:6789/0 conn(0x7f61941430c0 legacy=0x7f61941e1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:03.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f61897fa640 10 monclient: _finish_hunting 0 2024-03-25T18:16:03.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f61897fa640 1 monclient: found mon.b 2024-03-25T18:16:03.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f61897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:03.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f61897fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:16:03.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619414e070 con 0x7f619413f710 2024-03-25T18:16:03.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f61897fa640 10 monclient: _finish_auth 0 2024-03-25T18:16:03.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f61897fa640 10 monclient: _check_auth_tickets 2024-03-25T18:16:03.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f61897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:03.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f619ae25640 5 monclient: authenticate success, global_id 4384 2024-03-25T18:16:03.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f619ae25640 10 monclient: _renew_subs 2024-03-25T18:16:03.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f619ae25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:16:03.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f619ae25640 1 -- 172.21.15.63:0/1228164680 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61940b11e0 con 0x7f619413f710 2024-03-25T18:16:03.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f619ae25640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:03.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f619ae25640 10 monclient: _renew_subs 2024-03-25T18:16:03.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f619ae25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:16:03.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f619ae25640 1 -- 172.21.15.63:0/1228164680 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61940b1710 con 0x7f619413f710 2024-03-25T18:16:03.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.521+0000 7f619ae25640 1 librados: init done 2024-03-25T18:16:03.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.522+0000 7f619ae25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:03.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.522+0000 7f619ae25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:03.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.522+0000 7f619ae25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:16:03.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.522+0000 7f619ae25640 1 -- 172.21.15.63:0/1228164680 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619414e260 con 0x7f619413f710 2024-03-25T18:16:03.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.801+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6184003ac0 con 0x7f619413f710 2024-03-25T18:16:03.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.801+0000 7f61897fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:03.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.801+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6184003ed0 con 0x7f619413f710 2024-03-25T18:16:03.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.801+0000 7f61897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:03.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.801+0000 7f61897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:03.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.801+0000 7f61897fa640 10 monclient: dump: 2024-03-25T18:16:03.803 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:03.803 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:03.803 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:03.803 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:03.803 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:03.803 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:03.803 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:03.803 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:03.804 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:03.804 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:03.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.801+0000 7f618affd640 4 set_mon_vals no callback set 2024-03-25T18:16:03.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.802+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f61840040f0 con 0x7f619413f710 2024-03-25T18:16:03.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.803+0000 7f6198bc1640 10 monclient: get_auth_request con 0x7f618006de00 auth_method 0 2024-03-25T18:16:03.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.803+0000 7f61897fa640 10 client.4384.objecter ms_handle_connect 0x7f618006de00 2024-03-25T18:16:03.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.807+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f6184004420 con 0x7f619413f710 2024-03-25T18:16:03.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.807+0000 7f61897fa640 10 client.4384.objecter ms_dispatch 0x7f6194000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:03.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.807+0000 7f61897fa640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:03.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.807+0000 7f61897fa640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:03.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.807+0000 7f61897fa640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-25T18:16:03.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.808+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6184050550 con 0x7f619413f710 2024-03-25T18:16:03.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.808+0000 7f61897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:03.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:03.808+0000 7f61897fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:04.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.067+0000 7f619ae25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:04.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.067+0000 7f619ae25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:04.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.067+0000 7f619ae25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:16:04.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.067+0000 7f619ae25640 1 -- 172.21.15.63:0/1228164680 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61940627b0 con 0x7f619413f710 2024-03-25T18:16:04.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.093+0000 7f61897fa640 1 -- 172.21.15.63:0/1228164680 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f6184052480 con 0x7f619413f710 2024-03-25T18:16:04.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.093+0000 7f61897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:04.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.093+0000 7f61897fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:04.094 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:16:04.095 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:04.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.096+0000 7f619ae25640 10 librados: watch_flush enter 2024-03-25T18:16:04.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.096+0000 7f619ae25640 10 librados: watch_flush exit 2024-03-25T18:16:04.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.096+0000 7f619ae25640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-25T18:16:04.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.096+0000 7f619ae25640 10 client.4384.objecter successfully canceled tick 2024-03-25T18:16:04.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.096+0000 7f619ae25640 1 -- 172.21.15.63:0/1228164680 >> v1:172.21.15.81:6800/3640340873 conn(0x7f618006de00 legacy=0x7f61800702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.096+0000 7f619ae25640 10 monclient: shutdown 2024-03-25T18:16:04.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.096+0000 7f619ae25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:04.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.096+0000 7f619ae25640 1 -- 172.21.15.63:0/1228164680 >> v1:172.21.15.81:6789/0 conn(0x7f619413f710 legacy=0x7f61940ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.096+0000 7f619ae25640 1 -- 172.21.15.63:0/1228164680 shutdown_connections 2024-03-25T18:16:04.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.096+0000 7f619ae25640 1 -- 172.21.15.63:0/1228164680 >> 172.21.15.63:0/1228164680 conn(0x7f61940b7ea0 msgr2=0x7f6194074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:04.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.096+0000 7f619ae25640 1 -- 172.21.15.63:0/1228164680 shutdown_connections 2024-03-25T18:16:04.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.096+0000 7f619ae25640 1 -- 172.21.15.63:0/1228164680 wait complete. 2024-03-25T18:16:04.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:04.096+0000 7f619ae25640 1 librados: shutdown 2024-03-25T18:16:04.109 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:10.916 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:11.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.032+0000 7fdeba0dd640 10 monclient: get_monmap_and_config 2024-03-25T18:16:11.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.032+0000 7fdeba0dd640 10 monclient: build_initial_monmap 2024-03-25T18:16:11.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.032+0000 7fdeba0dd640 10 monclient: monmap: 2024-03-25T18:16:11.033 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:16:11.033 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:11.033 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:16:11.033527+0000 2024-03-25T18:16:11.033 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:16:11.033527+0000 2024-03-25T18:16:11.033 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:11.033 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:11.034 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:16:11.034 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:16:11.034 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:16:11.034 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:11.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.033+0000 7fdeba0dd640 1 Processor -- start 2024-03-25T18:16:11.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.033+0000 7fdeba0dd640 1 -- start start 2024-03-25T18:16:11.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.033+0000 7fdeba0dd640 10 monclient: init 2024-03-25T18:16:11.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.033+0000 7fdeba0dd640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:11.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.033+0000 7fdeba0dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:11.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.033+0000 7fdeba0dd640 10 monclient(hunting): picked mon.noname-a con 0x7fdeb40b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T18:16:11.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.034+0000 7fdeba0dd640 10 monclient(hunting): picked mon.noname-b con 0x7fdeb40b1730 addr v1:172.21.15.63:6790/0 2024-03-25T18:16:11.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.034+0000 7fdeba0dd640 10 monclient(hunting): picked mon.noname-c con 0x7fdeb414be90 addr v1:172.21.15.81:6789/0 2024-03-25T18:16:11.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.034+0000 7fdeba0dd640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb40af940 con 0x7fdeb40b4d60 2024-03-25T18:16:11.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.034+0000 7fdeba0dd640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb40b0a70 con 0x7fdeb414be90 2024-03-25T18:16:11.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.034+0000 7fdeba0dd640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb40b0d50 con 0x7fdeb40b1730 2024-03-25T18:16:11.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.034+0000 7fdeba0dd640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:11.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.034+0000 7fdeba0dd640 10 monclient(hunting): authenticate will time out at 7665.220423s 2024-03-25T18:16:11.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.044+0000 7fdeb3fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fdeb414be90 0x7fdeb414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:58932/0 (socket says 172.21.15.63:58932) 2024-03-25T18:16:11.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.044+0000 7fdeb3fff640 1 -- 172.21.15.63:0/1885694123 learned_addr learned my addr 172.21.15.63:0/1885694123 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:16:11.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.070+0000 7fdeb27fc640 1 -- 172.21.15.63:0/1885694123 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443065350 0 0) 0x7fdeb40b0a70 con 0x7fdeb414be90 2024-03-25T18:16:11.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.070+0000 7fdeb27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:11.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.070+0000 7fdeb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:11.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.070+0000 7fdeb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:11.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.070+0000 7fdeb27fc640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:16:11.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.070+0000 7fdeb27fc640 1 -- 172.21.15.63:0/1885694123 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde9c003520 con 0x7fdeb414be90 2024-03-25T18:16:11.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.073+0000 7fdeb27fc640 1 -- 172.21.15.63:0/1885694123 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576188098 0 0) 0x7fdeb40b0d50 con 0x7fdeb40b1730 2024-03-25T18:16:11.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.073+0000 7fdeb27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:11.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.073+0000 7fdeb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:11.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.073+0000 7fdeb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:11.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.073+0000 7fdeb27fc640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:16:11.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.073+0000 7fdeb27fc640 1 -- 172.21.15.63:0/1885694123 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeb40b0a70 con 0x7fdeb40b1730 2024-03-25T18:16:11.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.076+0000 7fdeb27fc640 1 -- 172.21.15.63:0/1885694123 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2625236584 0 0) 0x7fde9c003520 con 0x7fdeb414be90 2024-03-25T18:16:11.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.076+0000 7fdeb27fc640 1 -- 172.21.15.63:0/1885694123 >> v1:172.21.15.63:6790/0 conn(0x7fdeb40b1730 legacy=0x7fdeb40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:11.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.076+0000 7fdeb27fc640 1 -- 172.21.15.63:0/1885694123 >> v1:172.21.15.63:6789/0 conn(0x7fdeb40b4d60 legacy=0x7fdeb413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:11.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.076+0000 7fdeb27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:11.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.076+0000 7fdeb27fc640 1 monclient: found mon.noname-c 2024-03-25T18:16:11.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.076+0000 7fdeb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:11.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.076+0000 7fdeb27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:16:11.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.076+0000 7fdeb27fc640 1 -- 172.21.15.63:0/1885694123 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb40b1030 con 0x7fdeb414be90 2024-03-25T18:16:11.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.077+0000 7fdeb27fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:11.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.077+0000 7fdeb27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:11.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.077+0000 7fdeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:11.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.077+0000 7fdeb27fc640 1 -- 172.21.15.63:0/1885694123 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fdeac001a10 con 0x7fdeb414be90 2024-03-25T18:16:11.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.077+0000 7fdeb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:11.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.077+0000 7fdeb27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:11.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.077+0000 7fdeb27fc640 10 monclient: dump: 2024-03-25T18:16:11.079 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:11.079 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:11.079 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:11.079 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:11.079 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:11.080 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:11.080 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:11.080 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:11.080 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:11.080 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:11.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.077+0000 7fdeba0dd640 5 monclient: authenticate success, global_id 4387 2024-03-25T18:16:11.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.077+0000 7fdeba0dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:11.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.090+0000 7fdeb27fc640 1 -- 172.21.15.63:0/1885694123 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdeac001b70 con 0x7fdeb414be90 2024-03-25T18:16:11.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.090+0000 7fdeb27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:11.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.091+0000 7fdeb27fc640 1 -- 172.21.15.63:0/1885694123 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fdeac001f80 con 0x7fdeb414be90 2024-03-25T18:16:11.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.091+0000 7fdeba0dd640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:11.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.091+0000 7fdeba0dd640 4 set_mon_vals no callback set 2024-03-25T18:16:11.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.091+0000 7fdeb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:11.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.091+0000 7fdeb27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:11.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.091+0000 7fdeb27fc640 10 monclient: dump: 2024-03-25T18:16:11.092 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:11.092 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:11.092 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:11.092 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:11.092 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:11.092 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:11.092 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:11.092 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:11.093 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:11.093 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:11.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.091+0000 7fdeba0dd640 10 monclient: shutdown 2024-03-25T18:16:11.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.091+0000 7fdeba0dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:11.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.091+0000 7fdeba0dd640 1 -- 172.21.15.63:0/1885694123 >> v1:172.21.15.81:6789/0 conn(0x7fdeb414be90 legacy=0x7fdeb414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:11.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.092+0000 7fdeba0dd640 1 -- 172.21.15.63:0/1885694123 shutdown_connections 2024-03-25T18:16:11.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.092+0000 7fdeba0dd640 1 -- 172.21.15.63:0/1885694123 >> 172.21.15.63:0/1885694123 conn(0x7fdeb40617a0 msgr2=0x7fdeb4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:11.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.092+0000 7fdeba0dd640 1 -- 172.21.15.63:0/1885694123 shutdown_connections 2024-03-25T18:16:11.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.092+0000 7fdeba0dd640 1 -- 172.21.15.63:0/1885694123 wait complete. 2024-03-25T18:16:11.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.092+0000 7fdeba0dd640 10 monclient: build_initial_monmap 2024-03-25T18:16:11.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.092+0000 7fdeba0dd640 10 monclient: monmap: 2024-03-25T18:16:11.095 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:16:11.095 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:11.096 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:16:11.096 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:16:11.096 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:11.096 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:16:11.096 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:16:11.096 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:16:11.096 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:16:11.096 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:11.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 1 librados: starting msgr at 2024-03-25T18:16:11.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 1 librados: starting objecter 2024-03-25T18:16:11.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 1 Processor -- start 2024-03-25T18:16:11.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 1 -- start start 2024-03-25T18:16:11.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 1 librados: setting wanted keys 2024-03-25T18:16:11.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 1 librados: calling monclient init 2024-03-25T18:16:11.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 10 monclient: init 2024-03-25T18:16:11.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:11.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:11.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 10 monclient(hunting): picked mon.noname-c con 0x7fdeb40b1730 addr v1:172.21.15.63:6790/0 2024-03-25T18:16:11.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 10 monclient(hunting): picked mon.noname-a con 0x7fdeb40b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T18:16:11.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 10 monclient(hunting): picked mon.noname-b con 0x7fdeb414be90 addr v1:172.21.15.81:6789/0 2024-03-25T18:16:11.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb4143740 con 0x7fdeb40b4d60 2024-03-25T18:16:11.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb4143260 con 0x7fdeb414be90 2024-03-25T18:16:11.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb40aaa90 con 0x7fdeb40b1730 2024-03-25T18:16:11.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:11.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.093+0000 7fdeba0dd640 10 monclient(hunting): authenticate will time out at 7665.280204s 2024-03-25T18:16:11.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.102+0000 7fdeb2ffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fdeb40b4d60 0x7fdeb41d7890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:50510/0 (socket says 172.21.15.63:50510) 2024-03-25T18:16:11.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.102+0000 7fdeb2ffd640 1 -- 172.21.15.63:0/3896914879 learned_addr learned my addr 172.21.15.63:0/3896914879 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:16:11.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.107+0000 7fde9bfff640 10 client.?.objecter ms_handle_connect 0x7fdeb414be90 2024-03-25T18:16:11.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.107+0000 7fde9bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:11.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.107+0000 7fde9bfff640 10 client.?.objecter ms_handle_connect 0x7fdeb40b4d60 2024-03-25T18:16:11.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.107+0000 7fde9bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:11.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.115+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198807287 0 0) 0x7fdeb4143740 con 0x7fdeb40b4d60 2024-03-25T18:16:11.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.115+0000 7fde9bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:11.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.115+0000 7fde9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:11.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.115+0000 7fde9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:11.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.115+0000 7fde9bfff640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:16:11.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.115+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdea0004eb0 con 0x7fdeb40b4d60 2024-03-25T18:16:11.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.121+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065221094 0 0) 0x7fdeb4143260 con 0x7fdeb414be90 2024-03-25T18:16:11.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.121+0000 7fde9bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:11.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.121+0000 7fde9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:11.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.121+0000 7fde9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:11.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.121+0000 7fde9bfff640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:16:11.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.121+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeb4143740 con 0x7fdeb414be90 2024-03-25T18:16:11.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.123+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 296018160 0 0) 0x7fdea0004eb0 con 0x7fdeb40b4d60 2024-03-25T18:16:11.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.123+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdeb4143260 con 0x7fdeb40b4d60 2024-03-25T18:16:11.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.123+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fdea8001a10 con 0x7fdeb40b4d60 2024-03-25T18:16:11.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.123+0000 7fde9bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:11.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.123+0000 7fde9bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:11.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.123+0000 7fde9bfff640 10 monclient(hunting): dump: 2024-03-25T18:16:11.124 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:11.124 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:11.124 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:11.124 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:11.124 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:11.125 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:11.125 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:11.125 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:11.125 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:11.125 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:11.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.123+0000 7fde9bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:11.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.126+0000 7fde9bfff640 10 client.?.objecter ms_handle_connect 0x7fdeb40b1730 2024-03-25T18:16:11.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.126+0000 7fde9bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:11.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1623115019 0 0) 0x7fdeb4143260 con 0x7fdeb40b4d60 2024-03-25T18:16:11.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 >> v1:172.21.15.63:6790/0 conn(0x7fdeb40b1730 legacy=0x7fdeb4138010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:11.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 >> v1:172.21.15.81:6789/0 conn(0x7fdeb414be90 legacy=0x7fdeb41d7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:11.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fde9bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:11.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fde9bfff640 1 monclient: found mon.a 2024-03-25T18:16:11.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fde9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:11.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fde9bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:11.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb4138720 con 0x7fdeb40b4d60 2024-03-25T18:16:11.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fde9bfff640 10 monclient: _finish_auth 0 2024-03-25T18:16:11.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fde9bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:11.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fde9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:11.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fdeba0dd640 5 monclient: authenticate success, global_id 4374 2024-03-25T18:16:11.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fdeba0dd640 10 monclient: _renew_subs 2024-03-25T18:16:11.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fdeba0dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:11.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fdeba0dd640 1 -- 172.21.15.63:0/3896914879 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb4140380 con 0x7fdeb40b4d60 2024-03-25T18:16:11.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fdeba0dd640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:11.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fdeba0dd640 10 monclient: _renew_subs 2024-03-25T18:16:11.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fdeba0dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:11.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fdeba0dd640 1 -- 172.21.15.63:0/3896914879 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb4142a70 con 0x7fdeb40b4d60 2024-03-25T18:16:11.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.135+0000 7fdeba0dd640 1 librados: init done 2024-03-25T18:16:11.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.136+0000 7fdeba0dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:11.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.136+0000 7fdeba0dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:11.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.136+0000 7fdeba0dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:11.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.136+0000 7fdeba0dd640 1 -- 172.21.15.63:0/3896914879 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb4138990 con 0x7fdeb40b4d60 2024-03-25T18:16:11.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.440+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdea8001c50 con 0x7fdeb40b4d60 2024-03-25T18:16:11.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.440+0000 7fde9bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:11.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.440+0000 7fdeb1ffb640 4 set_mon_vals no callback set 2024-03-25T18:16:11.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.456+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fdea80038c0 con 0x7fdeb40b4d60 2024-03-25T18:16:11.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.456+0000 7fde9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:11.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.456+0000 7fde9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:11.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.456+0000 7fde9bfff640 10 monclient: dump: 2024-03-25T18:16:11.458 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:11.458 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:11.458 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:11.458 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:11.458 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:11.458 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:11.458 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:11.458 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:11.458 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:11.458 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.457+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fdea8003aa0 con 0x7fdeb40b4d60 2024-03-25T18:16:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.458+0000 7fdeb37fe640 10 monclient: get_auth_request con 0x7fdea006dcc0 auth_method 0 2024-03-25T18:16:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.458+0000 7fde9bfff640 10 client.4374.objecter ms_handle_connect 0x7fdea006dcc0 2024-03-25T18:16:11.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.463+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fdea80546f0 con 0x7fdeb40b4d60 2024-03-25T18:16:11.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.463+0000 7fde9bfff640 10 client.4374.objecter ms_dispatch 0x7fdeb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:11.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.463+0000 7fde9bfff640 3 client.4374.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:11.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.463+0000 7fde9bfff640 3 client.4374.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:11.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.463+0000 7fde9bfff640 20 client.4374.objecter dump_active .. 0 homeless 2024-03-25T18:16:11.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.463+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdea8050200 con 0x7fdeb40b4d60 2024-03-25T18:16:11.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.463+0000 7fde9bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:11.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.463+0000 7fde9bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:11.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.728+0000 7fdeba0dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:11.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.728+0000 7fdeba0dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:11.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.728+0000 7fdeba0dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:11.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.728+0000 7fdeba0dd640 1 -- 172.21.15.63:0/3896914879 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdeb40627b0 con 0x7fdeb40b4d60 2024-03-25T18:16:11.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.790+0000 7fde9bfff640 1 -- 172.21.15.63:0/3896914879 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fdea8052130 con 0x7fdeb40b4d60 2024-03-25T18:16:11.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.790+0000 7fde9bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:11.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.790+0000 7fde9bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:11.791 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:16:11.791 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:11.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.792+0000 7fdeba0dd640 10 librados: watch_flush enter 2024-03-25T18:16:11.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.792+0000 7fdeba0dd640 10 librados: watch_flush exit 2024-03-25T18:16:11.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.792+0000 7fdeba0dd640 20 client.4374.objecter shutdown clearing up homeless session... 2024-03-25T18:16:11.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.792+0000 7fdeba0dd640 10 client.4374.objecter successfully canceled tick 2024-03-25T18:16:11.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.792+0000 7fdeba0dd640 1 -- 172.21.15.63:0/3896914879 >> v1:172.21.15.81:6800/3640340873 conn(0x7fdea006dcc0 legacy=0x7fdea0070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:11.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.792+0000 7fdeba0dd640 10 monclient: shutdown 2024-03-25T18:16:11.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.792+0000 7fdeba0dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:11.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.792+0000 7fdeba0dd640 1 -- 172.21.15.63:0/3896914879 >> v1:172.21.15.63:6789/0 conn(0x7fdeb40b4d60 legacy=0x7fdeb41d7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:11.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.792+0000 7fdeba0dd640 1 -- 172.21.15.63:0/3896914879 shutdown_connections 2024-03-25T18:16:11.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.792+0000 7fdeba0dd640 1 -- 172.21.15.63:0/3896914879 >> 172.21.15.63:0/3896914879 conn(0x7fdeb40617a0 msgr2=0x7fdeb40b6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:11.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.792+0000 7fdeba0dd640 1 -- 172.21.15.63:0/3896914879 shutdown_connections 2024-03-25T18:16:11.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.792+0000 7fdeba0dd640 1 -- 172.21.15.63:0/3896914879 wait complete. 2024-03-25T18:16:11.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:11.792+0000 7fdeba0dd640 1 librados: shutdown 2024-03-25T18:16:11.803 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:18.596 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:04:00.288 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:18.610 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:18.650 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:04:00.342 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:18.687 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:04:00.379 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:18.702 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:04:00.363 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:18.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.702+0000 7f89aea50640 10 monclient: get_monmap_and_config 2024-03-25T18:16:18.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.702+0000 7f89aea50640 10 monclient: build_initial_monmap 2024-03-25T18:16:18.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.702+0000 7f89aea50640 10 monclient: monmap: 2024-03-25T18:16:18.703 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:16:18.703 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:18.703 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:16:18.703706+0000 2024-03-25T18:16:18.704 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:16:18.703706+0000 2024-03-25T18:16:18.704 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:18.704 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:18.704 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:16:18.704 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:16:18.704 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:16:18.704 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:18.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.703+0000 7f89aea50640 1 Processor -- start 2024-03-25T18:16:18.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.703+0000 7f89aea50640 1 -- start start 2024-03-25T18:16:18.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.703+0000 7f89aea50640 10 monclient: init 2024-03-25T18:16:18.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.704+0000 7f89aea50640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:18.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.704+0000 7f89aea50640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:18.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.704+0000 7f89aea50640 10 monclient(hunting): picked mon.noname-c con 0x7f89a80a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:16:18.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.704+0000 7f89aea50640 10 monclient(hunting): picked mon.noname-a con 0x7f89a813f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:16:18.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.704+0000 7f89aea50640 10 monclient(hunting): picked mon.noname-b con 0x7f89a81430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:16:18.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.704+0000 7f89aea50640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a8142520 con 0x7f89a813f710 2024-03-25T18:16:18.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.704+0000 7f89aea50640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a80b28a0 con 0x7f89a80a98f0 2024-03-25T18:16:18.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.704+0000 7f89aea50640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a80b2a30 con 0x7f89a81430c0 2024-03-25T18:16:18.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.704+0000 7f89aea50640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:18.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.704+0000 7f89aea50640 10 monclient(hunting): authenticate will time out at 7672.890520s 2024-03-25T18:16:18.711 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:04:00.372 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:18.722 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:04:00.415 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:18.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.723+0000 7f89a7fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f89a80a98f0 0x7f89a80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:33964/0 (socket says 172.21.15.63:33964) 2024-03-25T18:16:18.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.724+0000 7f89a7fff640 1 -- 172.21.15.63:0/3920255386 learned_addr learned my addr 172.21.15.63:0/3920255386 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:16:18.736 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:04:00.428 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:18.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.745+0000 7f89a6ffd640 1 -- 172.21.15.63:0/3920255386 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898608860 0 0) 0x7f89a80b28a0 con 0x7f89a80a98f0 2024-03-25T18:16:18.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.746+0000 7f89a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.746+0000 7f89a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.746+0000 7f89a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.746+0000 7f89a6ffd640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:16:18.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.746+0000 7f89a6ffd640 1 -- 172.21.15.63:0/3920255386 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8990003520 con 0x7f89a80a98f0 2024-03-25T18:16:18.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.754+0000 7f89a6ffd640 1 -- 172.21.15.63:0/3920255386 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807615373 0 0) 0x7f89a80b2a30 con 0x7f89a81430c0 2024-03-25T18:16:18.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.754+0000 7f89a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.754+0000 7f89a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.754+0000 7f89a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.754+0000 7f89a6ffd640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:16:18.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.754+0000 7f89a6ffd640 1 -- 172.21.15.63:0/3920255386 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a80b28a0 con 0x7f89a81430c0 2024-03-25T18:16:18.757 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:04:00.418 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:18.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.765+0000 7f89a6ffd640 1 -- 172.21.15.63:0/3920255386 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2382806152 0 0) 0x7f8990003520 con 0x7f89a80a98f0 2024-03-25T18:16:18.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89a6ffd640 1 -- 172.21.15.63:0/3920255386 >> v1:172.21.15.63:6790/0 conn(0x7f89a81430c0 legacy=0x7f89a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89a6ffd640 1 -- 172.21.15.63:0/3920255386 >> v1:172.21.15.63:6789/0 conn(0x7f89a813f710 legacy=0x7f89a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:16:18.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89a6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:18.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89a6ffd640 1 monclient: found mon.noname-c 2024-03-25T18:16:18.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:18.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89a6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:16:18.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89a6ffd640 1 -- 172.21.15.63:0/3920255386 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a80b44c0 con 0x7f89a80a98f0 2024-03-25T18:16:18.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89a6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:18.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89a6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:18.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:18.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89a6ffd640 1 -- 172.21.15.63:0/3920255386 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8998001690 con 0x7f89a80a98f0 2024-03-25T18:16:18.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89aea50640 5 monclient: authenticate success, global_id 4393 2024-03-25T18:16:18.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89aea50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:18.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:18.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:18.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89a6ffd640 10 monclient: dump: 2024-03-25T18:16:18.769 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:18.769 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:18.769 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:18.769 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:18.769 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:18.769 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:18.769 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:18.769 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:18.769 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:18.769 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:18.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.766+0000 7f89aea50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:18.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.777+0000 7f89a6ffd640 1 -- 172.21.15.63:0/3920255386 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89980017f0 con 0x7f89a80a98f0 2024-03-25T18:16:18.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.777+0000 7f89a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:18.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.777+0000 7f89a6ffd640 1 -- 172.21.15.63:0/3920255386 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8998001c00 con 0x7f89a80a98f0 2024-03-25T18:16:18.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.777+0000 7f89a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:18.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.777+0000 7f89a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:18.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.777+0000 7f89a6ffd640 10 monclient: dump: 2024-03-25T18:16:18.779 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:18.779 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:18.779 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:18.779 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:18.779 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:18.779 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:18.779 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:18.779 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:18.779 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:18.779 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:18.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.777+0000 7f89aea50640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:18.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.777+0000 7f89aea50640 4 set_mon_vals no callback set 2024-03-25T18:16:18.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.778+0000 7f89aea50640 10 monclient: shutdown 2024-03-25T18:16:18.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.778+0000 7f89aea50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:18.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.778+0000 7f89aea50640 1 -- 172.21.15.63:0/3920255386 >> v1:172.21.15.81:6789/0 conn(0x7f89a80a98f0 legacy=0x7f89a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.778+0000 7f89aea50640 1 -- 172.21.15.63:0/3920255386 shutdown_connections 2024-03-25T18:16:18.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.778+0000 7f89aea50640 1 -- 172.21.15.63:0/3920255386 >> 172.21.15.63:0/3920255386 conn(0x7f89a80b7ea0 msgr2=0x7f89a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:18.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.778+0000 7f89aea50640 1 -- 172.21.15.63:0/3920255386 shutdown_connections 2024-03-25T18:16:18.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.778+0000 7f89aea50640 1 -- 172.21.15.63:0/3920255386 wait complete. 2024-03-25T18:16:18.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.779+0000 7f89aea50640 10 monclient: build_initial_monmap 2024-03-25T18:16:18.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.779+0000 7f89aea50640 10 monclient: monmap: 2024-03-25T18:16:18.782 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:16:18.782 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:18.782 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:16:18.782 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:16:18.782 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:18.782 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:16:18.782 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:16:18.782 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:16:18.782 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:16:18.782 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:18.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.780+0000 7f89aea50640 1 librados: starting msgr at 2024-03-25T18:16:18.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.780+0000 7f89aea50640 1 librados: starting objecter 2024-03-25T18:16:18.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.780+0000 7f89aea50640 1 Processor -- start 2024-03-25T18:16:18.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.780+0000 7f89aea50640 1 -- start start 2024-03-25T18:16:18.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.780+0000 7f89aea50640 1 librados: setting wanted keys 2024-03-25T18:16:18.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.780+0000 7f89aea50640 1 librados: calling monclient init 2024-03-25T18:16:18.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.780+0000 7f89aea50640 10 monclient: init 2024-03-25T18:16:18.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.780+0000 7f89aea50640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:18.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.781+0000 7f89aea50640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:18.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.781+0000 7f89aea50640 10 monclient(hunting): picked mon.noname-a con 0x7f89a80a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:16:18.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.781+0000 7f89aea50640 10 monclient(hunting): picked mon.noname-c con 0x7f89a813f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:16:18.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.781+0000 7f89aea50640 10 monclient(hunting): picked mon.noname-b con 0x7f89a81430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:16:18.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.781+0000 7f89aea50640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a81d8400 con 0x7f89a80a98f0 2024-03-25T18:16:18.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.781+0000 7f89aea50640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a81d7df0 con 0x7f89a81430c0 2024-03-25T18:16:18.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.781+0000 7f89aea50640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a81d7f80 con 0x7f89a813f710 2024-03-25T18:16:18.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.781+0000 7f89aea50640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:18.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.781+0000 7f89aea50640 10 monclient(hunting): authenticate will time out at 7672.967397s 2024-03-25T18:16:18.788 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:04:00.480 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:18.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.826+0000 7f89acfed640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f89a81430c0 0x7f89a8135120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:33998/0 (socket says 172.21.15.63:33998) 2024-03-25T18:16:18.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.827+0000 7f89acfed640 1 -- 172.21.15.63:0/912941021 learned_addr learned my addr 172.21.15.63:0/912941021 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:16:18.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.829+0000 7f89a4ff9640 10 client.?.objecter ms_handle_connect 0x7f89a813f710 2024-03-25T18:16:18.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.829+0000 7f89a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:18.833 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.833+0000 7f89a4ff9640 10 client.?.objecter ms_handle_connect 0x7f89a81430c0 2024-03-25T18:16:18.834 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.833+0000 7f89a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:18.844 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.844+0000 7f89a4ff9640 10 client.?.objecter ms_handle_connect 0x7f89a80a98f0 2024-03-25T18:16:18.845 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.844+0000 7f89a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:18.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.861+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641799540 0 0) 0x7f89a81d7f80 con 0x7f89a813f710 2024-03-25T18:16:18.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.861+0000 7f89a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.861+0000 7f89a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.861+0000 7f89a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.861+0000 7f89a4ff9640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:16:18.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.861+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8994004eb0 con 0x7f89a813f710 2024-03-25T18:16:18.869 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.868+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085367166 0 0) 0x7f89a81d7df0 con 0x7f89a81430c0 2024-03-25T18:16:18.869 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.868+0000 7f89a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.869 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.868+0000 7f89a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.870 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.868+0000 7f89a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.870 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.868+0000 7f89a4ff9640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:16:18.870 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.868+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a81d7f80 con 0x7f89a81430c0 2024-03-25T18:16:18.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.870+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941445040 0 0) 0x7f89a81d8400 con 0x7f89a80a98f0 2024-03-25T18:16:18.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.870+0000 7f89a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.870+0000 7f89a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.870+0000 7f89a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.870+0000 7f89a4ff9640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:16:18.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.870+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a81d7df0 con 0x7f89a80a98f0 2024-03-25T18:16:18.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.875+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3835752969 0 0) 0x7f8994004eb0 con 0x7f89a813f710 2024-03-25T18:16:18.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.875+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89a81d8400 con 0x7f89a813f710 2024-03-25T18:16:18.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.875+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f899c001870 con 0x7f89a813f710 2024-03-25T18:16:18.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.875+0000 7f89a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:18.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.875+0000 7f89a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:18.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.875+0000 7f89a4ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:18.877 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:18.877 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:18.877 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:18.877 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:18.877 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:18.877 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:18.877 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:18.877 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:18.877 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:18.877 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:18.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.875+0000 7f89a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:18.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.882+0000 7f89a7fff640 1 -- 172.21.15.63:0/912941021 >> v1:172.21.15.63:6789/0 conn(0x7f89a80a98f0 legacy=0x7f89a81d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T18:16:18.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.882+0000 7f89a7fff640 1 -- 172.21.15.63:0/912941021 >> v1:172.21.15.63:6789/0 conn(0x7f89a80a98f0 legacy=0x7f89a81d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:16:18.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.882+0000 7f89a7fff640 1 --1- 172.21.15.63:0/912941021 >> v1:172.21.15.63:6789/0 conn(0x7f89a80a98f0 0x7f89a81d76e0 :-1 s=OPENED pgs=24109 cs=1 l=1).handle_message read tag failed 2024-03-25T18:16:18.884 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.882+0000 7f89a7fff640 1 --1- 172.21.15.63:0/912941021 >> v1:172.21.15.63:6789/0 conn(0x7f89a80a98f0 0x7f89a81d76e0 :-1 s=OPENED pgs=24109 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:16:18.884 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.882+0000 7f89a4ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.63:6789/0 2024-03-25T18:16:18.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.894+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 335984949 0 0) 0x7f89a81d7f80 con 0x7f89a81430c0 2024-03-25T18:16:18.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.894+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8994004eb0 con 0x7f89a81430c0 2024-03-25T18:16:18.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.894+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f89a0001c50 con 0x7f89a81430c0 2024-03-25T18:16:18.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.894+0000 7f89a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:18.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.894+0000 7f89a4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:18.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.894+0000 7f89a4ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:18.895 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:18.895 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:18.895 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:18.895 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:18.895 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:18.895 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:18.895 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:18.895 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:18.895 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:18.896 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:18.900 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2461336484 0 0) 0x7f89a81d8400 con 0x7f89a813f710 2024-03-25T18:16:18.900 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 >> v1:172.21.15.81:6789/0 conn(0x7f89a81430c0 legacy=0x7f89a8135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.900 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 >> v1:172.21.15.63:6789/0 conn(0x7f89a80a98f0 legacy=0x7f89a81d76e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:16:18.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89a4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:16:18.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89a4ff9640 1 monclient: found mon.c 2024-03-25T18:16:18.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:18.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89a4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:16:18.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a81d8110 con 0x7f89a813f710 2024-03-25T18:16:18.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89a4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:16:18.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89a4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:16:18.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:18.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89aea50640 5 monclient: authenticate success, global_id 4382 2024-03-25T18:16:18.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89aea50640 10 monclient: _renew_subs 2024-03-25T18:16:18.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89aea50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:16:18.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89aea50640 1 -- 172.21.15.63:0/912941021 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a80b7300 con 0x7f89a813f710 2024-03-25T18:16:18.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89aea50640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:18.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89aea50640 10 monclient: _renew_subs 2024-03-25T18:16:18.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89aea50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:16:18.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89aea50640 1 -- 172.21.15.63:0/912941021 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a8133880 con 0x7f89a813f710 2024-03-25T18:16:18.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.899+0000 7f89aea50640 1 librados: init done 2024-03-25T18:16:18.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.900+0000 7f89aea50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:18.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.900+0000 7f89aea50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:18.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.900+0000 7f89aea50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:16:18.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:18.900+0000 7f89aea50640 1 -- 172.21.15.63:0/912941021 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a80b3ef0 con 0x7f89a813f710 2024-03-25T18:16:19.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.234+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f899c0035f0 con 0x7f89a813f710 2024-03-25T18:16:19.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.234+0000 7f89a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:19.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.234+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f899c001d80 con 0x7f89a813f710 2024-03-25T18:16:19.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.234+0000 7f89a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.234+0000 7f89a4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:19.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.234+0000 7f89a4ff9640 10 monclient: dump: 2024-03-25T18:16:19.236 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:19.236 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:19.236 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:19.236 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:19.236 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.236 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:19.236 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:19.237 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:19.237 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:19.237 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:19.237 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.234+0000 7f89a67fc640 4 set_mon_vals no callback set 2024-03-25T18:16:19.237 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.235+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f899c019830 con 0x7f89a813f710 2024-03-25T18:16:19.237 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.236+0000 7f89a7fff640 10 monclient: get_auth_request con 0x7f899406df10 auth_method 0 2024-03-25T18:16:19.237 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.236+0000 7f89a4ff9640 10 client.4382.objecter ms_handle_connect 0x7f899406df10 2024-03-25T18:16:19.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.245+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f899c053540 con 0x7f89a813f710 2024-03-25T18:16:19.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.245+0000 7f89a4ff9640 10 client.4382.objecter ms_dispatch 0x7f89a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:19.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.245+0000 7f89a4ff9640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:19.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.245+0000 7f89a4ff9640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:19.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.245+0000 7f89a4ff9640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-25T18:16:19.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.245+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f899c04ee80 con 0x7f89a813f710 2024-03-25T18:16:19.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.245+0000 7f89a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:19.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.245+0000 7f89a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:19.500 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.499+0000 7f89aea50640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:19.500 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.499+0000 7f89aea50640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:19.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.499+0000 7f89aea50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:16:19.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.499+0000 7f89aea50640 1 -- 172.21.15.63:0/912941021 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89a80b76a0 con 0x7f89a813f710 2024-03-25T18:16:19.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.520+0000 7f89a4ff9640 1 -- 172.21.15.63:0/912941021 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f899c056ba0 con 0x7f89a813f710 2024-03-25T18:16:19.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.520+0000 7f89a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:19.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.520+0000 7f89a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:19.521 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:16:19.521 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:19.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.522+0000 7f89aea50640 10 librados: watch_flush enter 2024-03-25T18:16:19.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.522+0000 7f89aea50640 10 librados: watch_flush exit 2024-03-25T18:16:19.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.522+0000 7f89aea50640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-25T18:16:19.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.522+0000 7f89aea50640 10 client.4382.objecter successfully canceled tick 2024-03-25T18:16:19.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.522+0000 7f89aea50640 1 -- 172.21.15.63:0/912941021 >> v1:172.21.15.81:6800/3640340873 conn(0x7f899406df10 legacy=0x7f89940703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.522+0000 7f89aea50640 10 monclient: shutdown 2024-03-25T18:16:19.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.522+0000 7f89aea50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:19.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.522+0000 7f89aea50640 1 -- 172.21.15.63:0/912941021 >> v1:172.21.15.63:6790/0 conn(0x7f89a813f710 legacy=0x7f89a8132f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.522+0000 7f89aea50640 1 -- 172.21.15.63:0/912941021 shutdown_connections 2024-03-25T18:16:19.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.522+0000 7f89aea50640 1 -- 172.21.15.63:0/912941021 >> 172.21.15.63:0/912941021 conn(0x7f89a80b7ea0 msgr2=0x7f89a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:19.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.522+0000 7f89aea50640 1 -- 172.21.15.63:0/912941021 shutdown_connections 2024-03-25T18:16:19.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.522+0000 7f89aea50640 1 -- 172.21.15.63:0/912941021 wait complete. 2024-03-25T18:16:19.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:19.522+0000 7f89aea50640 1 librados: shutdown 2024-03-25T18:16:19.535 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:26.343 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:26.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.450+0000 7f8a1f20e640 10 monclient: get_monmap_and_config 2024-03-25T18:16:26.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.450+0000 7f8a1f20e640 10 monclient: build_initial_monmap 2024-03-25T18:16:26.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.450+0000 7f8a1f20e640 10 monclient: monmap: 2024-03-25T18:16:26.452 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:16:26.452 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:26.452 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:16:26.452086+0000 2024-03-25T18:16:26.452 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:16:26.452086+0000 2024-03-25T18:16:26.452 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:26.452 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:26.452 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:16:26.452 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:16:26.452 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:16:26.452 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:26.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.451+0000 7f8a1f20e640 1 Processor -- start 2024-03-25T18:16:26.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.451+0000 7f8a1f20e640 1 -- start start 2024-03-25T18:16:26.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.451+0000 7f8a1f20e640 10 monclient: init 2024-03-25T18:16:26.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.452+0000 7f8a1f20e640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:26.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.452+0000 7f8a1f20e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:26.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.452+0000 7f8a1f20e640 10 monclient(hunting): picked mon.noname-a con 0x7f8a180a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:16:26.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.452+0000 7f8a1f20e640 10 monclient(hunting): picked mon.noname-c con 0x7f8a1813f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:16:26.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.452+0000 7f8a1f20e640 10 monclient(hunting): picked mon.noname-b con 0x7f8a181430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:16:26.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.452+0000 7f8a1f20e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a18142520 con 0x7f8a180a98f0 2024-03-25T18:16:26.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.452+0000 7f8a1f20e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a180b28a0 con 0x7f8a1813f710 2024-03-25T18:16:26.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.452+0000 7f8a1f20e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a180b2a30 con 0x7f8a181430c0 2024-03-25T18:16:26.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.452+0000 7f8a1f20e640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:26.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.452+0000 7f8a1f20e640 10 monclient(hunting): authenticate will time out at 7680.638758s 2024-03-25T18:16:26.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.483+0000 7f8a17fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f8a1813f710 0x7f8a181429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:39100/0 (socket says 172.21.15.63:39100) 2024-03-25T18:16:26.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.483+0000 7f8a17fff640 1 -- 172.21.15.63:0/4098500939 learned_addr learned my addr 172.21.15.63:0/4098500939 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:16:26.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.508+0000 7f8a177fe640 1 -- 172.21.15.63:0/4098500939 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368644062 0 0) 0x7f8a180b2a30 con 0x7f8a181430c0 2024-03-25T18:16:26.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.508+0000 7f8a177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:26.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.508+0000 7f8a177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:26.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.508+0000 7f8a177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:26.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.508+0000 7f8a177fe640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:16:26.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.508+0000 7f8a177fe640 1 -- 172.21.15.63:0/4098500939 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a04003520 con 0x7f8a181430c0 2024-03-25T18:16:26.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.509+0000 7f8a177fe640 1 -- 172.21.15.63:0/4098500939 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947497197 0 0) 0x7f8a180b28a0 con 0x7f8a1813f710 2024-03-25T18:16:26.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.509+0000 7f8a177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:26.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.509+0000 7f8a177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:26.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.509+0000 7f8a177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:26.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.509+0000 7f8a177fe640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:16:26.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.509+0000 7f8a177fe640 1 -- 172.21.15.63:0/4098500939 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a180b2a30 con 0x7f8a1813f710 2024-03-25T18:16:26.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.523+0000 7f8a177fe640 1 -- 172.21.15.63:0/4098500939 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917948873 0 0) 0x7f8a18142520 con 0x7f8a180a98f0 2024-03-25T18:16:26.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.523+0000 7f8a177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:26.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.523+0000 7f8a177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:26.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.523+0000 7f8a177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:26.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.523+0000 7f8a177fe640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:16:26.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.523+0000 7f8a177fe640 1 -- 172.21.15.63:0/4098500939 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a180b28a0 con 0x7f8a180a98f0 2024-03-25T18:16:26.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.525+0000 7f8a177fe640 1 -- 172.21.15.63:0/4098500939 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1603485528 0 0) 0x7f8a180b2a30 con 0x7f8a1813f710 2024-03-25T18:16:26.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.525+0000 7f8a177fe640 1 -- 172.21.15.63:0/4098500939 >> v1:172.21.15.63:6790/0 conn(0x7f8a181430c0 legacy=0x7f8a1814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:26.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.525+0000 7f8a177fe640 1 -- 172.21.15.63:0/4098500939 >> v1:172.21.15.63:6789/0 conn(0x7f8a180a98f0 legacy=0x7f8a180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:26.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.525+0000 7f8a177fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:26.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.525+0000 7f8a177fe640 1 monclient: found mon.noname-c 2024-03-25T18:16:26.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.525+0000 7f8a177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:26.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.525+0000 7f8a177fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:16:26.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.525+0000 7f8a177fe640 1 -- 172.21.15.63:0/4098500939 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a180b44c0 con 0x7f8a1813f710 2024-03-25T18:16:26.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.525+0000 7f8a177fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:26.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.525+0000 7f8a177fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:26.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.525+0000 7f8a177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:26.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.525+0000 7f8a177fe640 1 -- 172.21.15.63:0/4098500939 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8a08001670 con 0x7f8a1813f710 2024-03-25T18:16:26.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.526+0000 7f8a1f20e640 5 monclient: authenticate success, global_id 4399 2024-03-25T18:16:26.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.526+0000 7f8a1f20e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:26.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.526+0000 7f8a177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:26.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.526+0000 7f8a177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:26.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.526+0000 7f8a177fe640 10 monclient: dump: 2024-03-25T18:16:26.528 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:26.528 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:26.528 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:26.528 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:26.528 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:26.528 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:26.528 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:26.529 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:26.529 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:26.529 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:26.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.526+0000 7f8a1f20e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:26.539 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.538+0000 7f8a177fe640 1 -- 172.21.15.63:0/4098500939 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a08001890 con 0x7f8a1813f710 2024-03-25T18:16:26.539 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.538+0000 7f8a177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:26.539 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.538+0000 7f8a177fe640 1 -- 172.21.15.63:0/4098500939 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8a08001c20 con 0x7f8a1813f710 2024-03-25T18:16:26.539 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.538+0000 7f8a177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:26.539 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.538+0000 7f8a177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:26.539 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.538+0000 7f8a177fe640 10 monclient: dump: 2024-03-25T18:16:26.540 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:26.540 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:26.540 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:26.540 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:26.540 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:26.540 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:26.540 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:26.540 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:26.540 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:26.540 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:26.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.538+0000 7f8a1f20e640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:26.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.538+0000 7f8a1f20e640 4 set_mon_vals no callback set 2024-03-25T18:16:26.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.539+0000 7f8a1f20e640 10 monclient: shutdown 2024-03-25T18:16:26.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.539+0000 7f8a1f20e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:26.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.539+0000 7f8a1f20e640 1 -- 172.21.15.63:0/4098500939 >> v1:172.21.15.81:6789/0 conn(0x7f8a1813f710 legacy=0x7f8a181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:26.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.539+0000 7f8a1f20e640 1 -- 172.21.15.63:0/4098500939 shutdown_connections 2024-03-25T18:16:26.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.539+0000 7f8a1f20e640 1 -- 172.21.15.63:0/4098500939 >> 172.21.15.63:0/4098500939 conn(0x7f8a180b7ea0 msgr2=0x7f8a180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:26.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.539+0000 7f8a1f20e640 1 -- 172.21.15.63:0/4098500939 shutdown_connections 2024-03-25T18:16:26.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.539+0000 7f8a1f20e640 1 -- 172.21.15.63:0/4098500939 wait complete. 2024-03-25T18:16:26.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.540+0000 7f8a1f20e640 10 monclient: build_initial_monmap 2024-03-25T18:16:26.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.540+0000 7f8a1f20e640 10 monclient: monmap: 2024-03-25T18:16:26.542 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:16:26.542 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:26.542 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:16:26.543 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:16:26.543 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:26.543 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:16:26.543 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:16:26.543 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:16:26.543 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:16:26.543 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:26.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.541+0000 7f8a1f20e640 1 librados: starting msgr at 2024-03-25T18:16:26.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.541+0000 7f8a1f20e640 1 librados: starting objecter 2024-03-25T18:16:26.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.541+0000 7f8a1f20e640 1 Processor -- start 2024-03-25T18:16:26.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.541+0000 7f8a1f20e640 1 -- start start 2024-03-25T18:16:26.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.541+0000 7f8a1f20e640 1 librados: setting wanted keys 2024-03-25T18:16:26.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.541+0000 7f8a1f20e640 1 librados: calling monclient init 2024-03-25T18:16:26.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.541+0000 7f8a1f20e640 10 monclient: init 2024-03-25T18:16:26.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.542+0000 7f8a1f20e640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:26.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.542+0000 7f8a1f20e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:26.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.542+0000 7f8a1f20e640 10 monclient(hunting): picked mon.noname-c con 0x7f8a180a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:16:26.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.542+0000 7f8a1f20e640 10 monclient(hunting): picked mon.noname-a con 0x7f8a1813f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:16:26.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.542+0000 7f8a1f20e640 10 monclient(hunting): picked mon.noname-b con 0x7f8a181430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:16:26.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.542+0000 7f8a1f20e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a181d7a90 con 0x7f8a1813f710 2024-03-25T18:16:26.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.542+0000 7f8a1f20e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a180b1700 con 0x7f8a181430c0 2024-03-25T18:16:26.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.542+0000 7f8a1f20e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a180b13d0 con 0x7f8a180a98f0 2024-03-25T18:16:26.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.542+0000 7f8a1f20e640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:26.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.542+0000 7f8a1f20e640 10 monclient(hunting): authenticate will time out at 7680.728515s 2024-03-25T18:16:26.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.559+0000 7f8a1d7ab640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f8a181430c0 0x7f8a1814d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:39148/0 (socket says 172.21.15.63:39148) 2024-03-25T18:16:26.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.559+0000 7f8a1d7ab640 1 -- 172.21.15.63:0/2567663284 learned_addr learned my addr 172.21.15.63:0/2567663284 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:16:26.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.559+0000 7f8a157fa640 10 client.?.objecter ms_handle_connect 0x7f8a181430c0 2024-03-25T18:16:26.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.559+0000 7f8a157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:26.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.561+0000 7f8a157fa640 10 client.?.objecter ms_handle_connect 0x7f8a1813f710 2024-03-25T18:16:26.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.561+0000 7f8a157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:26.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.567+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930922371 0 0) 0x7f8a180b1700 con 0x7f8a181430c0 2024-03-25T18:16:26.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.567+0000 7f8a157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:26.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.567+0000 7f8a157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:26.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.567+0000 7f8a157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:26.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.567+0000 7f8a157fa640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:16:26.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.567+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89f8004eb0 con 0x7f8a181430c0 2024-03-25T18:16:26.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.575+0000 7f8a157fa640 10 client.?.objecter ms_handle_connect 0x7f8a180a98f0 2024-03-25T18:16:26.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.575+0000 7f8a157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:26.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.583+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027357491 0 0) 0x7f8a180b13d0 con 0x7f8a180a98f0 2024-03-25T18:16:26.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.583+0000 7f8a157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:26.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.583+0000 7f8a157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:26.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.583+0000 7f8a157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:26.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.583+0000 7f8a157fa640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:16:26.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.583+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a180b1700 con 0x7f8a180a98f0 2024-03-25T18:16:26.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.586+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907872882 0 0) 0x7f8a181d7a90 con 0x7f8a1813f710 2024-03-25T18:16:26.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.586+0000 7f8a157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:26.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.586+0000 7f8a157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:26.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.586+0000 7f8a157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:26.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.586+0000 7f8a157fa640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:16:26.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.586+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a180b13d0 con 0x7f8a1813f710 2024-03-25T18:16:26.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.589+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2401518479 0 0) 0x7f89f8004eb0 con 0x7f8a181430c0 2024-03-25T18:16:26.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.589+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a181d7a90 con 0x7f8a181430c0 2024-03-25T18:16:26.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.589+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8a000018b0 con 0x7f8a181430c0 2024-03-25T18:16:26.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.589+0000 7f8a157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:26.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.589+0000 7f8a157fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:26.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.589+0000 7f8a157fa640 10 monclient(hunting): dump: 2024-03-25T18:16:26.590 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:26.591 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:26.591 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:26.591 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:26.591 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:26.591 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:26.591 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:26.591 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:26.591 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:26.591 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:26.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.589+0000 7f8a157fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:26.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.596+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3848803311 0 0) 0x7f8a180b1700 con 0x7f8a180a98f0 2024-03-25T18:16:26.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.596+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89f8004eb0 con 0x7f8a180a98f0 2024-03-25T18:16:26.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.596+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8a10001bf0 con 0x7f8a180a98f0 2024-03-25T18:16:26.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.596+0000 7f8a157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:26.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.596+0000 7f8a157fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:26.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.596+0000 7f8a157fa640 10 monclient(hunting): dump: 2024-03-25T18:16:26.597 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:26.597 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:26.597 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:26.597 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:26.598 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:26.598 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:26.598 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:26.598 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:26.598 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:26.598 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:26.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.597+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1211228575 0 0) 0x7f8a181d7a90 con 0x7f8a181430c0 2024-03-25T18:16:26.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.597+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 >> v1:172.21.15.63:6790/0 conn(0x7f8a180a98f0 legacy=0x7f8a181d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:26.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.597+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 >> v1:172.21.15.63:6789/0 conn(0x7f8a1813f710 legacy=0x7f8a181d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:26.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.597+0000 7f8a157fa640 10 monclient: _finish_hunting 0 2024-03-25T18:16:26.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.597+0000 7f8a157fa640 1 monclient: found mon.b 2024-03-25T18:16:26.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.597+0000 7f8a157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:26.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.597+0000 7f8a157fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:16:26.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.597+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1814eb50 con 0x7f8a181430c0 2024-03-25T18:16:26.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.597+0000 7f8a157fa640 10 monclient: _finish_auth 0 2024-03-25T18:16:26.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.597+0000 7f8a157fa640 10 monclient: _check_auth_tickets 2024-03-25T18:16:26.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.597+0000 7f8a157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:26.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.597+0000 7f8a1f20e640 5 monclient: authenticate success, global_id 4402 2024-03-25T18:16:26.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.597+0000 7f8a1f20e640 10 monclient: _renew_subs 2024-03-25T18:16:26.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.597+0000 7f8a1f20e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:16:26.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.597+0000 7f8a1f20e640 1 -- 172.21.15.63:0/2567663284 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a1814dc50 con 0x7f8a181430c0 2024-03-25T18:16:26.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.598+0000 7f8a1f20e640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:26.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.598+0000 7f8a1f20e640 10 monclient: _renew_subs 2024-03-25T18:16:26.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.598+0000 7f8a1f20e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:16:26.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.598+0000 7f8a1f20e640 1 -- 172.21.15.63:0/2567663284 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a1814e290 con 0x7f8a181430c0 2024-03-25T18:16:26.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.598+0000 7f8a1f20e640 1 librados: init done 2024-03-25T18:16:26.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.599+0000 7f8a1f20e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:26.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.599+0000 7f8a1f20e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:26.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.599+0000 7f8a1f20e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:16:26.603 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.599+0000 7f8a1f20e640 1 -- 172.21.15.63:0/2567663284 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a180b1560 con 0x7f8a181430c0 2024-03-25T18:16:26.870 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.869+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a00001d60 con 0x7f8a181430c0 2024-03-25T18:16:26.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.869+0000 7f8a157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:26.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.869+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8a00003610 con 0x7f8a181430c0 2024-03-25T18:16:26.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.869+0000 7f8a157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:26.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.869+0000 7f8a157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:26.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.869+0000 7f8a157fa640 10 monclient: dump: 2024-03-25T18:16:26.871 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:26.871 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:26.871 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:26.871 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:26.871 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:26.871 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:26.871 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:26.871 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:26.871 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:26.871 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:26.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.869+0000 7f8a16ffd640 4 set_mon_vals no callback set 2024-03-25T18:16:26.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.870+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f8a00003830 con 0x7f8a181430c0 2024-03-25T18:16:26.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.871+0000 7f8a1cfaa640 10 monclient: get_auth_request con 0x7f89f806df70 auth_method 0 2024-03-25T18:16:26.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.871+0000 7f8a157fa640 10 client.4402.objecter ms_handle_connect 0x7f89f806df70 2024-03-25T18:16:26.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.880+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f8a00054590 con 0x7f8a181430c0 2024-03-25T18:16:26.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.880+0000 7f8a157fa640 10 client.4402.objecter ms_dispatch 0x7f8a18000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:26.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.880+0000 7f8a157fa640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:26.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.880+0000 7f8a157fa640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:26.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.880+0000 7f8a157fa640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T18:16:26.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.880+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a0004fed0 con 0x7f8a181430c0 2024-03-25T18:16:26.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.880+0000 7f8a157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:26.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:26.880+0000 7f8a157fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:27.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.136+0000 7f8a1f20e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:27.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.136+0000 7f8a1f20e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:27.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.136+0000 7f8a1f20e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:16:27.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.136+0000 7f8a1f20e640 1 -- 172.21.15.63:0/2567663284 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a180b0d00 con 0x7f8a181430c0 2024-03-25T18:16:27.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.158+0000 7f8a157fa640 1 -- 172.21.15.63:0/2567663284 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f8a00051fd0 con 0x7f8a181430c0 2024-03-25T18:16:27.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.158+0000 7f8a157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:27.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.158+0000 7f8a157fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:27.160 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:16:27.160 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:27.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.161+0000 7f8a1f20e640 10 librados: watch_flush enter 2024-03-25T18:16:27.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.161+0000 7f8a1f20e640 10 librados: watch_flush exit 2024-03-25T18:16:27.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.161+0000 7f8a1f20e640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T18:16:27.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.161+0000 7f8a1f20e640 10 client.4402.objecter successfully canceled tick 2024-03-25T18:16:27.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.161+0000 7f8a1f20e640 1 -- 172.21.15.63:0/2567663284 >> v1:172.21.15.81:6800/3640340873 conn(0x7f89f806df70 legacy=0x7f89f8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.161+0000 7f8a1f20e640 10 monclient: shutdown 2024-03-25T18:16:27.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.161+0000 7f8a1f20e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:27.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.161+0000 7f8a1f20e640 1 -- 172.21.15.63:0/2567663284 >> v1:172.21.15.81:6789/0 conn(0x7f8a181430c0 legacy=0x7f8a1814d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.161+0000 7f8a1f20e640 1 -- 172.21.15.63:0/2567663284 shutdown_connections 2024-03-25T18:16:27.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.161+0000 7f8a1f20e640 1 -- 172.21.15.63:0/2567663284 >> 172.21.15.63:0/2567663284 conn(0x7f8a180b7ea0 msgr2=0x7f8a18074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:27.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.161+0000 7f8a1f20e640 1 -- 172.21.15.63:0/2567663284 shutdown_connections 2024-03-25T18:16:27.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.161+0000 7f8a1f20e640 1 -- 172.21.15.63:0/2567663284 wait complete. 2024-03-25T18:16:27.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:27.161+0000 7f8a1f20e640 1 librados: shutdown 2024-03-25T18:16:27.174 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:33.980 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:34.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.097+0000 7ffadc014640 10 monclient: get_monmap_and_config 2024-03-25T18:16:34.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.097+0000 7ffadc014640 10 monclient: build_initial_monmap 2024-03-25T18:16:34.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.097+0000 7ffadc014640 10 monclient: monmap: 2024-03-25T18:16:34.099 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:16:34.099 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:34.099 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:16:34.098894+0000 2024-03-25T18:16:34.099 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:16:34.098894+0000 2024-03-25T18:16:34.099 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:34.099 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:34.099 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:16:34.100 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:16:34.100 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:16:34.100 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:34.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.098+0000 7ffadc014640 1 Processor -- start 2024-03-25T18:16:34.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.098+0000 7ffadc014640 1 -- start start 2024-03-25T18:16:34.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.098+0000 7ffadc014640 10 monclient: init 2024-03-25T18:16:34.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.099+0000 7ffadc014640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:34.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.099+0000 7ffadc014640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:34.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.099+0000 7ffadc014640 10 monclient(hunting): picked mon.noname-b con 0x7ffad40a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:16:34.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.099+0000 7ffadc014640 10 monclient(hunting): picked mon.noname-c con 0x7ffad413f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:16:34.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.099+0000 7ffadc014640 10 monclient(hunting): picked mon.noname-a con 0x7ffad41430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:16:34.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.099+0000 7ffadc014640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad4142520 con 0x7ffad41430c0 2024-03-25T18:16:34.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.099+0000 7ffadc014640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad40b28a0 con 0x7ffad413f710 2024-03-25T18:16:34.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.099+0000 7ffadc014640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad40b2a30 con 0x7ffad40a98f0 2024-03-25T18:16:34.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.099+0000 7ffadc014640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:34.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.099+0000 7ffadc014640 10 monclient(hunting): authenticate will time out at 7688.286266s 2024-03-25T18:16:34.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.115+0000 7ffada5b1640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7ffad41430c0 0x7ffad414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56246/0 (socket says 172.21.15.63:56246) 2024-03-25T18:16:34.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.115+0000 7ffada5b1640 1 -- 172.21.15.63:0/997823306 learned_addr learned my addr 172.21.15.63:0/997823306 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:16:34.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.131+0000 7ffad8dae640 1 -- 172.21.15.63:0/997823306 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175779774 0 0) 0x7ffad40b2a30 con 0x7ffad40a98f0 2024-03-25T18:16:34.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.131+0000 7ffad8dae640 10 monclient(hunting): _negotiate 2024-03-25T18:16:34.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.131+0000 7ffad8dae640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:34.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.131+0000 7ffad8dae640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:34.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.131+0000 7ffad8dae640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:16:34.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.131+0000 7ffad8dae640 1 -- 172.21.15.63:0/997823306 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffac0003520 con 0x7ffad40a98f0 2024-03-25T18:16:34.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.131+0000 7ffad8dae640 1 -- 172.21.15.63:0/997823306 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962528502 0 0) 0x7ffad40b28a0 con 0x7ffad413f710 2024-03-25T18:16:34.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.131+0000 7ffad8dae640 10 monclient(hunting): _negotiate 2024-03-25T18:16:34.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.131+0000 7ffad8dae640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:34.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.131+0000 7ffad8dae640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:34.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.131+0000 7ffad8dae640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:16:34.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.131+0000 7ffad8dae640 1 -- 172.21.15.63:0/997823306 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad40b2a30 con 0x7ffad413f710 2024-03-25T18:16:34.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.134+0000 7ffad8dae640 1 -- 172.21.15.63:0/997823306 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123856206 0 0) 0x7ffad4142520 con 0x7ffad41430c0 2024-03-25T18:16:34.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.134+0000 7ffad8dae640 10 monclient(hunting): _negotiate 2024-03-25T18:16:34.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.134+0000 7ffad8dae640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:34.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.134+0000 7ffad8dae640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:34.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.134+0000 7ffad8dae640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:16:34.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.134+0000 7ffad8dae640 1 -- 172.21.15.63:0/997823306 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad40b28a0 con 0x7ffad41430c0 2024-03-25T18:16:34.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.141+0000 7ffad8dae640 1 -- 172.21.15.63:0/997823306 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1976324661 0 0) 0x7ffac0003520 con 0x7ffad40a98f0 2024-03-25T18:16:34.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.141+0000 7ffad8dae640 1 -- 172.21.15.63:0/997823306 >> v1:172.21.15.81:6789/0 conn(0x7ffad413f710 legacy=0x7ffad41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.141+0000 7ffad8dae640 1 -- 172.21.15.63:0/997823306 >> v1:172.21.15.63:6789/0 conn(0x7ffad41430c0 legacy=0x7ffad414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.141+0000 7ffad8dae640 10 monclient: _finish_hunting 0 2024-03-25T18:16:34.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.141+0000 7ffad8dae640 1 monclient: found mon.noname-b 2024-03-25T18:16:34.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.141+0000 7ffad8dae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:34.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.141+0000 7ffad8dae640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:16:34.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.141+0000 7ffad8dae640 1 -- 172.21.15.63:0/997823306 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad40b44c0 con 0x7ffad40a98f0 2024-03-25T18:16:34.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.141+0000 7ffad8dae640 10 monclient: _finish_auth 0 2024-03-25T18:16:34.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.141+0000 7ffad8dae640 10 monclient: _check_auth_tickets 2024-03-25T18:16:34.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.141+0000 7ffad8dae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:34.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.141+0000 7ffadc014640 5 monclient: authenticate success, global_id 4391 2024-03-25T18:16:34.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.141+0000 7ffadc014640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:34.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.143+0000 7ffad8dae640 1 -- 172.21.15.63:0/997823306 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ffad0001650 con 0x7ffad40a98f0 2024-03-25T18:16:34.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.143+0000 7ffad8dae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:34.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.143+0000 7ffad8dae640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:34.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.143+0000 7ffad8dae640 10 monclient: dump: 2024-03-25T18:16:34.144 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:34.144 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:34.144 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:34.144 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:34.144 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:34.144 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:34.144 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:34.145 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:34.145 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:34.145 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:34.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.143+0000 7ffadc014640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:34.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.144+0000 7ffad8dae640 1 -- 172.21.15.63:0/997823306 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffad00018a0 con 0x7ffad40a98f0 2024-03-25T18:16:34.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.144+0000 7ffad8dae640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:34.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.144+0000 7ffad8dae640 1 -- 172.21.15.63:0/997823306 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ffad0001c30 con 0x7ffad40a98f0 2024-03-25T18:16:34.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.144+0000 7ffad8dae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:34.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.144+0000 7ffad8dae640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:34.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.144+0000 7ffad8dae640 10 monclient: dump: 2024-03-25T18:16:34.147 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:34.147 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:34.148 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:34.148 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:34.148 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:34.148 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:34.148 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:34.148 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:34.148 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:34.148 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:34.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.144+0000 7ffadc014640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:34.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.144+0000 7ffadc014640 4 set_mon_vals no callback set 2024-03-25T18:16:34.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.145+0000 7ffadc014640 10 monclient: shutdown 2024-03-25T18:16:34.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.145+0000 7ffadc014640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:34.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.145+0000 7ffadc014640 1 -- 172.21.15.63:0/997823306 >> v1:172.21.15.63:6790/0 conn(0x7ffad40a98f0 legacy=0x7ffad40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.145+0000 7ffadc014640 1 -- 172.21.15.63:0/997823306 shutdown_connections 2024-03-25T18:16:34.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.145+0000 7ffadc014640 1 -- 172.21.15.63:0/997823306 >> 172.21.15.63:0/997823306 conn(0x7ffad40b7ea0 msgr2=0x7ffad40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:34.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.145+0000 7ffadc014640 1 -- 172.21.15.63:0/997823306 shutdown_connections 2024-03-25T18:16:34.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.145+0000 7ffadc014640 1 -- 172.21.15.63:0/997823306 wait complete. 2024-03-25T18:16:34.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.146+0000 7ffadc014640 10 monclient: build_initial_monmap 2024-03-25T18:16:34.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.146+0000 7ffadc014640 10 monclient: monmap: 2024-03-25T18:16:34.150 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:16:34.151 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:34.151 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:16:34.151 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:16:34.151 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:34.151 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:16:34.151 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:16:34.151 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:16:34.151 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:16:34.151 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:34.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.146+0000 7ffadc014640 1 librados: starting msgr at 2024-03-25T18:16:34.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.146+0000 7ffadc014640 1 librados: starting objecter 2024-03-25T18:16:34.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.147+0000 7ffadc014640 1 Processor -- start 2024-03-25T18:16:34.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.147+0000 7ffadc014640 1 -- start start 2024-03-25T18:16:34.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.147+0000 7ffadc014640 1 librados: setting wanted keys 2024-03-25T18:16:34.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.147+0000 7ffadc014640 1 librados: calling monclient init 2024-03-25T18:16:34.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.147+0000 7ffadc014640 10 monclient: init 2024-03-25T18:16:34.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.148+0000 7ffadc014640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:34.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.148+0000 7ffadc014640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:34.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.148+0000 7ffadc014640 10 monclient(hunting): picked mon.noname-a con 0x7ffad40a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:16:34.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.148+0000 7ffadc014640 10 monclient(hunting): picked mon.noname-c con 0x7ffad413f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:16:34.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.148+0000 7ffadc014640 10 monclient(hunting): picked mon.noname-b con 0x7ffad41430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:16:34.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.148+0000 7ffadc014640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad4136ec0 con 0x7ffad40a98f0 2024-03-25T18:16:34.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.148+0000 7ffadc014640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad41355f0 con 0x7ffad41430c0 2024-03-25T18:16:34.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.148+0000 7ffadc014640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad41319b0 con 0x7ffad413f710 2024-03-25T18:16:34.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.148+0000 7ffadc014640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:34.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.148+0000 7ffadc014640 10 monclient(hunting): authenticate will time out at 7688.334448s 2024-03-25T18:16:34.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.174+0000 7ffada5b1640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ffad41430c0 0x7ffad41312a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:43044/0 (socket says 172.21.15.63:43044) 2024-03-25T18:16:34.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.174+0000 7ffada5b1640 1 -- 172.21.15.63:0/241876641 learned_addr learned my addr 172.21.15.63:0/241876641 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:16:34.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.175+0000 7ffaca7fc640 10 client.?.objecter ms_handle_connect 0x7ffad41430c0 2024-03-25T18:16:34.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.175+0000 7ffaca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:34.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.179+0000 7ffaca7fc640 10 client.?.objecter ms_handle_connect 0x7ffad413f710 2024-03-25T18:16:34.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.179+0000 7ffaca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:34.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.180+0000 7ffaca7fc640 10 client.?.objecter ms_handle_connect 0x7ffad40a98f0 2024-03-25T18:16:34.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.180+0000 7ffaca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:34.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.187+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119442038 0 0) 0x7ffad4136ec0 con 0x7ffad40a98f0 2024-03-25T18:16:34.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.187+0000 7ffaca7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:34.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.187+0000 7ffaca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:34.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.187+0000 7ffaca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:34.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.187+0000 7ffaca7fc640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:16:34.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.187+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab8004eb0 con 0x7ffad40a98f0 2024-03-25T18:16:34.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.188+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19565266 0 0) 0x7ffad41319b0 con 0x7ffad413f710 2024-03-25T18:16:34.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.188+0000 7ffaca7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:34.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.188+0000 7ffaca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:34.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.188+0000 7ffaca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:34.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.188+0000 7ffaca7fc640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:16:34.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.188+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad4136ec0 con 0x7ffad413f710 2024-03-25T18:16:34.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.191+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246105802 0 0) 0x7ffad41355f0 con 0x7ffad41430c0 2024-03-25T18:16:34.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.191+0000 7ffaca7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:34.192 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.191+0000 7ffaca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:34.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.191+0000 7ffaca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:34.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.191+0000 7ffaca7fc640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:16:34.193 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.191+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad41319b0 con 0x7ffad41430c0 2024-03-25T18:16:34.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.194+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2594356139 0 0) 0x7ffab8004eb0 con 0x7ffad40a98f0 2024-03-25T18:16:34.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.194+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffad41355f0 con 0x7ffad40a98f0 2024-03-25T18:16:34.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.194+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ffad0001bb0 con 0x7ffad40a98f0 2024-03-25T18:16:34.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.194+0000 7ffaca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:34.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.194+0000 7ffaca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:34.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.194+0000 7ffaca7fc640 10 monclient(hunting): dump: 2024-03-25T18:16:34.196 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:34.196 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:34.196 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:34.196 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:34.196 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:34.196 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:34.196 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:34.196 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:34.196 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:34.197 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:34.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.194+0000 7ffaca7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:34.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.201+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2992549646 0 0) 0x7ffad41355f0 con 0x7ffad40a98f0 2024-03-25T18:16:34.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.201+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 >> v1:172.21.15.63:6790/0 conn(0x7ffad413f710 legacy=0x7ffad4130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.201+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 >> v1:172.21.15.81:6789/0 conn(0x7ffad41430c0 legacy=0x7ffad41312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.201+0000 7ffaca7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:34.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.201+0000 7ffaca7fc640 1 monclient: found mon.a 2024-03-25T18:16:34.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.201+0000 7ffaca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:34.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.201+0000 7ffaca7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:34.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.201+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad4135d80 con 0x7ffad40a98f0 2024-03-25T18:16:34.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.201+0000 7ffaca7fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:34.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.201+0000 7ffaca7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:34.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.201+0000 7ffaca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:34.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.201+0000 7ffadc014640 5 monclient: authenticate success, global_id 4389 2024-03-25T18:16:34.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.201+0000 7ffadc014640 10 monclient: _renew_subs 2024-03-25T18:16:34.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.201+0000 7ffadc014640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:34.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.201+0000 7ffadc014640 1 -- 172.21.15.63:0/241876641 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad4135f70 con 0x7ffad40a98f0 2024-03-25T18:16:34.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.202+0000 7ffadc014640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:34.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.202+0000 7ffadc014640 10 monclient: _renew_subs 2024-03-25T18:16:34.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.202+0000 7ffadc014640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:34.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.202+0000 7ffadc014640 1 -- 172.21.15.63:0/241876641 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad413b780 con 0x7ffad40a98f0 2024-03-25T18:16:34.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.202+0000 7ffadc014640 1 librados: init done 2024-03-25T18:16:34.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.202+0000 7ffadc014640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:34.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.203+0000 7ffadc014640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:34.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.203+0000 7ffadc014640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:34.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.203+0000 7ffadc014640 1 -- 172.21.15.63:0/241876641 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad40627b0 con 0x7ffad40a98f0 2024-03-25T18:16:34.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.514+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffad0003960 con 0x7ffad40a98f0 2024-03-25T18:16:34.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.514+0000 7ffaca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:34.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.514+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ffad0003ad0 con 0x7ffad40a98f0 2024-03-25T18:16:34.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.514+0000 7ffaca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:34.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.514+0000 7ffaca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:34.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.514+0000 7ffaca7fc640 10 monclient: dump: 2024-03-25T18:16:34.516 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:34.516 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:34.516 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:34.516 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:34.516 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:34.516 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:34.516 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:34.516 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:34.516 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:34.516 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:34.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.514+0000 7ffacbfff640 4 set_mon_vals no callback set 2024-03-25T18:16:34.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.515+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7ffad0019aa0 con 0x7ffad40a98f0 2024-03-25T18:16:34.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.515+0000 7ffad95af640 10 monclient: get_auth_request con 0x7ffab806ddf0 auth_method 0 2024-03-25T18:16:34.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.516+0000 7ffaca7fc640 10 client.4389.objecter ms_handle_connect 0x7ffab806ddf0 2024-03-25T18:16:34.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.521+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7ffad00534f0 con 0x7ffad40a98f0 2024-03-25T18:16:34.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.521+0000 7ffaca7fc640 10 client.4389.objecter ms_dispatch 0x7ffad4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:34.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.521+0000 7ffaca7fc640 3 client.4389.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:34.522 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.521+0000 7ffaca7fc640 3 client.4389.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:34.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.521+0000 7ffaca7fc640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-25T18:16:34.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.521+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffad004f000 con 0x7ffad40a98f0 2024-03-25T18:16:34.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.521+0000 7ffaca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:34.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.521+0000 7ffaca7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:34.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.760+0000 7ffadc014640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:34.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.760+0000 7ffadc014640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:34.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.760+0000 7ffadc014640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:34.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.760+0000 7ffadc014640 1 -- 172.21.15.63:0/241876641 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffad413e230 con 0x7ffad40a98f0 2024-03-25T18:16:34.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.794+0000 7ffaca7fc640 1 -- 172.21.15.63:0/241876641 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7ffad0050f30 con 0x7ffad40a98f0 2024-03-25T18:16:34.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.794+0000 7ffaca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:34.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.794+0000 7ffaca7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:34.795 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:16:34.795 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:34.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.797+0000 7ffadc014640 10 librados: watch_flush enter 2024-03-25T18:16:34.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.797+0000 7ffadc014640 10 librados: watch_flush exit 2024-03-25T18:16:34.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.797+0000 7ffadc014640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-25T18:16:34.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.797+0000 7ffadc014640 10 client.4389.objecter successfully canceled tick 2024-03-25T18:16:34.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.797+0000 7ffadc014640 1 -- 172.21.15.63:0/241876641 >> v1:172.21.15.81:6800/3640340873 conn(0x7ffab806ddf0 legacy=0x7ffab80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.797+0000 7ffadc014640 10 monclient: shutdown 2024-03-25T18:16:34.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.797+0000 7ffadc014640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:34.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.797+0000 7ffadc014640 1 -- 172.21.15.63:0/241876641 >> v1:172.21.15.63:6789/0 conn(0x7ffad40a98f0 legacy=0x7ffad41dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.797+0000 7ffadc014640 1 -- 172.21.15.63:0/241876641 shutdown_connections 2024-03-25T18:16:34.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.797+0000 7ffadc014640 1 -- 172.21.15.63:0/241876641 >> 172.21.15.63:0/241876641 conn(0x7ffad40b7ea0 msgr2=0x7ffad4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:34.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.797+0000 7ffadc014640 1 -- 172.21.15.63:0/241876641 shutdown_connections 2024-03-25T18:16:34.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.797+0000 7ffadc014640 1 -- 172.21.15.63:0/241876641 wait complete. 2024-03-25T18:16:34.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:34.797+0000 7ffadc014640 1 librados: shutdown 2024-03-25T18:16:34.811 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:41.618 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:41.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.729+0000 7f5312172640 10 monclient: get_monmap_and_config 2024-03-25T18:16:41.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.729+0000 7f5312172640 10 monclient: build_initial_monmap 2024-03-25T18:16:41.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.730+0000 7f5312172640 10 monclient: monmap: 2024-03-25T18:16:41.731 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:16:41.731 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:41.731 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:16:41.731137+0000 2024-03-25T18:16:41.731 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:16:41.731137+0000 2024-03-25T18:16:41.731 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:41.732 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:41.732 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:16:41.732 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:16:41.732 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:16:41.732 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:41.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.731+0000 7f5312172640 1 Processor -- start 2024-03-25T18:16:41.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.731+0000 7f5312172640 1 -- start start 2024-03-25T18:16:41.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.731+0000 7f5312172640 10 monclient: init 2024-03-25T18:16:41.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.731+0000 7f5312172640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:41.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.731+0000 7f5312172640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:41.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.731+0000 7f5312172640 10 monclient(hunting): picked mon.noname-a con 0x7f530c0a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:16:41.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.731+0000 7f5312172640 10 monclient(hunting): picked mon.noname-b con 0x7f530c13f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:16:41.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.731+0000 7f5312172640 10 monclient(hunting): picked mon.noname-c con 0x7f530c1430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:16:41.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.731+0000 7f5312172640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c142520 con 0x7f530c0a98f0 2024-03-25T18:16:41.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.731+0000 7f5312172640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c0b28a0 con 0x7f530c1430c0 2024-03-25T18:16:41.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.731+0000 7f5312172640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c0b2a30 con 0x7f530c13f710 2024-03-25T18:16:41.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.731+0000 7f5312172640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:41.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.731+0000 7f5312172640 10 monclient(hunting): authenticate will time out at 7695.918223s 2024-03-25T18:16:41.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.744+0000 7f530bfff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f530c1430c0 0x7f530c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:38730/0 (socket says 172.21.15.63:38730) 2024-03-25T18:16:41.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.744+0000 7f530bfff640 1 -- 172.21.15.63:0/203368437 learned_addr learned my addr 172.21.15.63:0/203368437 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:16:41.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.762+0000 7f530a7fc640 1 -- 172.21.15.63:0/203368437 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944030918 0 0) 0x7f530c0b28a0 con 0x7f530c1430c0 2024-03-25T18:16:41.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.762+0000 7f530a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:41.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.762+0000 7f530a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:41.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.762+0000 7f530a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:41.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.762+0000 7f530a7fc640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:16:41.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.762+0000 7f530a7fc640 1 -- 172.21.15.63:0/203368437 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52f4003520 con 0x7f530c1430c0 2024-03-25T18:16:41.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f530a7fc640 1 -- 172.21.15.63:0/203368437 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2219018180 0 0) 0x7f52f4003520 con 0x7f530c1430c0 2024-03-25T18:16:41.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f530a7fc640 1 -- 172.21.15.63:0/203368437 >> v1:172.21.15.63:6790/0 conn(0x7f530c13f710 legacy=0x7f530c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:41.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f530a7fc640 1 -- 172.21.15.63:0/203368437 >> v1:172.21.15.63:6789/0 conn(0x7f530c0a98f0 legacy=0x7f530c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:41.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f530a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:41.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f530a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:16:41.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f530a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:41.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f530a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:16:41.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f530a7fc640 1 -- 172.21.15.63:0/203368437 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c0b44c0 con 0x7f530c1430c0 2024-03-25T18:16:41.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f530a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:41.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f530a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:41.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f530a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:41.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f530a7fc640 1 -- 172.21.15.63:0/203368437 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5304001a10 con 0x7f530c1430c0 2024-03-25T18:16:41.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f5312172640 5 monclient: authenticate success, global_id 4411 2024-03-25T18:16:41.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f5312172640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:41.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f530a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:41.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f530a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:41.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f530a7fc640 10 monclient: dump: 2024-03-25T18:16:41.774 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:41.775 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:41.775 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:41.775 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:41.775 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:41.775 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:41.775 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:41.775 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:41.775 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:41.775 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:41.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.772+0000 7f5312172640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:41.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.781+0000 7f530a7fc640 1 -- 172.21.15.63:0/203368437 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5304001b70 con 0x7f530c1430c0 2024-03-25T18:16:41.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.781+0000 7f530a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:41.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.781+0000 7f530a7fc640 1 -- 172.21.15.63:0/203368437 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5304001f80 con 0x7f530c1430c0 2024-03-25T18:16:41.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.781+0000 7f530a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:41.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.781+0000 7f530a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:41.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.781+0000 7f530a7fc640 10 monclient: dump: 2024-03-25T18:16:41.783 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:41.783 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:41.783 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:41.783 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:41.783 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:41.783 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:41.783 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:41.783 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:41.783 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:41.784 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:41.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.781+0000 7f5312172640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:41.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.781+0000 7f5312172640 4 set_mon_vals no callback set 2024-03-25T18:16:41.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.782+0000 7f5312172640 10 monclient: shutdown 2024-03-25T18:16:41.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.782+0000 7f5312172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:41.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.782+0000 7f5312172640 1 -- 172.21.15.63:0/203368437 >> v1:172.21.15.81:6789/0 conn(0x7f530c1430c0 legacy=0x7f530c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:41.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.782+0000 7f5312172640 1 -- 172.21.15.63:0/203368437 shutdown_connections 2024-03-25T18:16:41.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.782+0000 7f5312172640 1 -- 172.21.15.63:0/203368437 >> 172.21.15.63:0/203368437 conn(0x7f530c0b7ea0 msgr2=0x7f530c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:41.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.782+0000 7f5312172640 1 -- 172.21.15.63:0/203368437 shutdown_connections 2024-03-25T18:16:41.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.782+0000 7f5312172640 1 -- 172.21.15.63:0/203368437 wait complete. 2024-03-25T18:16:41.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.783+0000 7f5312172640 10 monclient: build_initial_monmap 2024-03-25T18:16:41.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.783+0000 7f5312172640 10 monclient: monmap: 2024-03-25T18:16:41.787 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:16:41.787 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:41.787 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:16:41.787 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:16:41.787 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:41.787 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:16:41.787 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:16:41.787 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:16:41.787 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:16:41.787 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:41.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.783+0000 7f5312172640 1 librados: starting msgr at 2024-03-25T18:16:41.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.783+0000 7f5312172640 1 librados: starting objecter 2024-03-25T18:16:41.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.784+0000 7f5312172640 1 Processor -- start 2024-03-25T18:16:41.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.784+0000 7f5312172640 1 -- start start 2024-03-25T18:16:41.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.784+0000 7f5312172640 1 librados: setting wanted keys 2024-03-25T18:16:41.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.784+0000 7f5312172640 1 librados: calling monclient init 2024-03-25T18:16:41.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.784+0000 7f5312172640 10 monclient: init 2024-03-25T18:16:41.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.784+0000 7f5312172640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:41.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.785+0000 7f5312172640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:41.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.785+0000 7f5312172640 10 monclient(hunting): picked mon.noname-c con 0x7f530c0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:16:41.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.785+0000 7f5312172640 10 monclient(hunting): picked mon.noname-b con 0x7f530c13f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:16:41.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.785+0000 7f5312172640 10 monclient(hunting): picked mon.noname-a con 0x7f530c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:16:41.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.785+0000 7f5312172640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c136ad0 con 0x7f530c1430c0 2024-03-25T18:16:41.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.785+0000 7f5312172640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c0b7520 con 0x7f530c13f710 2024-03-25T18:16:41.788 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.785+0000 7f5312172640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c135810 con 0x7f530c0a98f0 2024-03-25T18:16:41.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.785+0000 7f5312172640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:41.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.785+0000 7f5312172640 10 monclient(hunting): authenticate will time out at 7695.971515s 2024-03-25T18:16:41.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.790+0000 7f530b7fe640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f530c0a98f0 0x7f530c135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:33476/0 (socket says 172.21.15.63:33476) 2024-03-25T18:16:41.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.790+0000 7f530b7fe640 1 -- 172.21.15.63:0/1305841846 learned_addr learned my addr 172.21.15.63:0/1305841846 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:16:41.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.792+0000 7f52f3fff640 10 client.?.objecter ms_handle_connect 0x7f530c0a98f0 2024-03-25T18:16:41.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.792+0000 7f52f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:41.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.793+0000 7f52f3fff640 10 client.?.objecter ms_handle_connect 0x7f530c13f710 2024-03-25T18:16:41.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.793+0000 7f52f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:41.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.860+0000 7f52f3fff640 10 client.?.objecter ms_handle_connect 0x7f530c1430c0 2024-03-25T18:16:41.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.860+0000 7f52f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:41.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.877+0000 7f52f3fff640 1 -- 172.21.15.63:0/1305841846 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344681066 0 0) 0x7f530c136ad0 con 0x7f530c1430c0 2024-03-25T18:16:41.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.877+0000 7f52f3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:41.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.877+0000 7f52f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:41.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.877+0000 7f52f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:41.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.877+0000 7f52f3fff640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:16:41.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.877+0000 7f52f3fff640 1 -- 172.21.15.63:0/1305841846 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52f8004eb0 con 0x7f530c1430c0 2024-03-25T18:16:41.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.896+0000 7f52f3fff640 1 -- 172.21.15.63:0/1305841846 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2418157979 0 0) 0x7f52f8004eb0 con 0x7f530c1430c0 2024-03-25T18:16:41.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.896+0000 7f52f3fff640 1 -- 172.21.15.63:0/1305841846 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f530c136ad0 con 0x7f530c1430c0 2024-03-25T18:16:41.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.896+0000 7f52f3fff640 1 -- 172.21.15.63:0/1305841846 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5304001e80 con 0x7f530c1430c0 2024-03-25T18:16:41.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.896+0000 7f52f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:41.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.896+0000 7f52f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:41.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.896+0000 7f52f3fff640 10 monclient(hunting): dump: 2024-03-25T18:16:41.898 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:41.898 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:41.898 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:41.898 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:41.898 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:41.898 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:41.898 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:41.898 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:41.898 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:41.898 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:41.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.896+0000 7f52f3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:41.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f52f3fff640 1 -- 172.21.15.63:0/1305841846 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3670491669 0 0) 0x7f530c136ad0 con 0x7f530c1430c0 2024-03-25T18:16:41.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f52f3fff640 1 -- 172.21.15.63:0/1305841846 >> v1:172.21.15.63:6790/0 conn(0x7f530c0a98f0 legacy=0x7f530c135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:41.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f52f3fff640 1 -- 172.21.15.63:0/1305841846 >> v1:172.21.15.81:6789/0 conn(0x7f530c13f710 legacy=0x7f530c0b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:41.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f52f3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:41.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f52f3fff640 1 monclient: found mon.a 2024-03-25T18:16:41.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f52f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:41.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f52f3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:41.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f52f3fff640 1 -- 172.21.15.63:0/1305841846 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c0b6180 con 0x7f530c1430c0 2024-03-25T18:16:41.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f52f3fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:41.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f52f3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:41.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f52f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:41.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f5312172640 5 monclient: authenticate success, global_id 4395 2024-03-25T18:16:41.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f5312172640 10 monclient: _renew_subs 2024-03-25T18:16:41.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f5312172640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:41.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f5312172640 1 -- 172.21.15.63:0/1305841846 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530c0b7a40 con 0x7f530c1430c0 2024-03-25T18:16:41.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f5312172640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:41.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f5312172640 10 monclient: _renew_subs 2024-03-25T18:16:41.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f5312172640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:41.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f5312172640 1 -- 172.21.15.63:0/1305841846 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f530c131840 con 0x7f530c1430c0 2024-03-25T18:16:41.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.916+0000 7f5312172640 1 librados: init done 2024-03-25T18:16:41.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.917+0000 7f5312172640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:41.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.917+0000 7f5312172640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:41.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.917+0000 7f5312172640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:41.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:41.917+0000 7f5312172640 1 -- 172.21.15.63:0/1305841846 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530c136f60 con 0x7f530c1430c0 2024-03-25T18:16:42.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.214+0000 7f52f3fff640 1 -- 172.21.15.63:0/1305841846 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5304004e10 con 0x7f530c1430c0 2024-03-25T18:16:42.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.214+0000 7f52f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:42.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.214+0000 7f52f3fff640 1 -- 172.21.15.63:0/1305841846 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5304004f80 con 0x7f530c1430c0 2024-03-25T18:16:42.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.214+0000 7f52f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:42.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.214+0000 7f52f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:42.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.214+0000 7f52f3fff640 10 monclient: dump: 2024-03-25T18:16:42.216 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:42.216 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:42.216 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:42.216 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:42.216 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:42.216 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:42.216 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:42.216 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:42.216 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:42.216 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:42.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.214+0000 7f5309ffb640 4 set_mon_vals no callback set 2024-03-25T18:16:42.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.215+0000 7f52f3fff640 1 -- 172.21.15.63:0/1305841846 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f530401af50 con 0x7f530c1430c0 2024-03-25T18:16:42.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.215+0000 7f530b7fe640 10 monclient: get_auth_request con 0x7f52f806dc90 auth_method 0 2024-03-25T18:16:42.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.216+0000 7f52f3fff640 10 client.4395.objecter ms_handle_connect 0x7f52f806dc90 2024-03-25T18:16:42.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.222+0000 7f52f3fff640 1 -- 172.21.15.63:0/1305841846 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f53040549e0 con 0x7f530c1430c0 2024-03-25T18:16:42.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.222+0000 7f52f3fff640 10 client.4395.objecter ms_dispatch 0x7f530c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:42.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.222+0000 7f52f3fff640 3 client.4395.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:42.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.222+0000 7f52f3fff640 3 client.4395.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:42.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.222+0000 7f52f3fff640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-25T18:16:42.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.222+0000 7f52f3fff640 1 -- 172.21.15.63:0/1305841846 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53040504f0 con 0x7f530c1430c0 2024-03-25T18:16:42.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.222+0000 7f52f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:42.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.222+0000 7f52f3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:42.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.458+0000 7f5312172640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:42.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.458+0000 7f5312172640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:42.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.458+0000 7f5312172640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:42.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.458+0000 7f5312172640 1 -- 172.21.15.63:0/1305841846 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f530c0b6310 con 0x7f530c1430c0 2024-03-25T18:16:42.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.509+0000 7f52f3fff640 1 -- 172.21.15.63:0/1305841846 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f5304052420 con 0x7f530c1430c0 2024-03-25T18:16:42.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.509+0000 7f52f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:42.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.509+0000 7f52f3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:42.511 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:16:42.511 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:42.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.512+0000 7f5312172640 10 librados: watch_flush enter 2024-03-25T18:16:42.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.512+0000 7f5312172640 10 librados: watch_flush exit 2024-03-25T18:16:42.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.512+0000 7f5312172640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-25T18:16:42.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.512+0000 7f5312172640 10 client.4395.objecter successfully canceled tick 2024-03-25T18:16:42.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.512+0000 7f5312172640 1 -- 172.21.15.63:0/1305841846 >> v1:172.21.15.81:6800/3640340873 conn(0x7f52f806dc90 legacy=0x7f52f8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:42.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.512+0000 7f5312172640 10 monclient: shutdown 2024-03-25T18:16:42.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.512+0000 7f5312172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:42.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.512+0000 7f5312172640 1 -- 172.21.15.63:0/1305841846 >> v1:172.21.15.63:6789/0 conn(0x7f530c1430c0 legacy=0x7f530c130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:42.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.512+0000 7f5312172640 1 -- 172.21.15.63:0/1305841846 shutdown_connections 2024-03-25T18:16:42.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.512+0000 7f5312172640 1 -- 172.21.15.63:0/1305841846 >> 172.21.15.63:0/1305841846 conn(0x7f530c0b7ea0 msgr2=0x7f530c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:42.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.512+0000 7f5312172640 1 -- 172.21.15.63:0/1305841846 shutdown_connections 2024-03-25T18:16:42.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.512+0000 7f5312172640 1 -- 172.21.15.63:0/1305841846 wait complete. 2024-03-25T18:16:42.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:42.512+0000 7f5312172640 1 librados: shutdown 2024-03-25T18:16:42.525 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:49.331 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:49.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.447+0000 7f96a3972640 10 monclient: get_monmap_and_config 2024-03-25T18:16:49.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.447+0000 7f96a3972640 10 monclient: build_initial_monmap 2024-03-25T18:16:49.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.447+0000 7f96a3972640 10 monclient: monmap: 2024-03-25T18:16:49.447 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:16:49.448 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:49.448 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:16:49.447267+0000 2024-03-25T18:16:49.448 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:16:49.447267+0000 2024-03-25T18:16:49.448 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:49.448 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:49.448 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:16:49.448 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:16:49.448 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:16:49.448 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:49.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.447+0000 7f96a3972640 1 Processor -- start 2024-03-25T18:16:49.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.447+0000 7f96a3972640 1 -- start start 2024-03-25T18:16:49.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.448+0000 7f96a3972640 10 monclient: init 2024-03-25T18:16:49.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.448+0000 7f96a3972640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:49.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.448+0000 7f96a3972640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:49.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.448+0000 7f96a3972640 10 monclient(hunting): picked mon.noname-c con 0x7f969c0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:16:49.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.448+0000 7f96a3972640 10 monclient(hunting): picked mon.noname-b con 0x7f969c13f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:16:49.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.448+0000 7f96a3972640 10 monclient(hunting): picked mon.noname-a con 0x7f969c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:16:49.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.448+0000 7f96a3972640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c142520 con 0x7f969c1430c0 2024-03-25T18:16:49.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.448+0000 7f96a3972640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c0b28a0 con 0x7f969c0a98f0 2024-03-25T18:16:49.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.448+0000 7f96a3972640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c0b2a30 con 0x7f969c13f710 2024-03-25T18:16:49.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.448+0000 7f96a3972640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:49.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.448+0000 7f96a3972640 10 monclient(hunting): authenticate will time out at 7703.634191s 2024-03-25T18:16:49.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.458+0000 7f96a0f0d640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f969c13f710 0x7f969c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:43790/0 (socket says 172.21.15.63:43790) 2024-03-25T18:16:49.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.458+0000 7f96a0f0d640 1 -- 172.21.15.63:0/2178743470 learned_addr learned my addr 172.21.15.63:0/2178743470 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:16:49.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.473+0000 7f9693fff640 1 -- 172.21.15.63:0/2178743470 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560486541 0 0) 0x7f969c0b28a0 con 0x7f969c0a98f0 2024-03-25T18:16:49.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.473+0000 7f9693fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:49.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.473+0000 7f9693fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:49.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.473+0000 7f9693fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:49.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.473+0000 7f9693fff640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:16:49.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.473+0000 7f9693fff640 1 -- 172.21.15.63:0/2178743470 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9688003520 con 0x7f969c0a98f0 2024-03-25T18:16:49.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.485+0000 7f9693fff640 1 -- 172.21.15.63:0/2178743470 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604900813 0 0) 0x7f969c0b2a30 con 0x7f969c13f710 2024-03-25T18:16:49.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.485+0000 7f9693fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:49.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.485+0000 7f9693fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:49.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.485+0000 7f9693fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:49.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.485+0000 7f9693fff640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:16:49.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.485+0000 7f9693fff640 1 -- 172.21.15.63:0/2178743470 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f969c0b28a0 con 0x7f969c13f710 2024-03-25T18:16:49.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 1 -- 172.21.15.63:0/2178743470 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1369923010 0 0) 0x7f969c142520 con 0x7f969c1430c0 2024-03-25T18:16:49.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:49.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:49.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:49.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:16:49.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 1 -- 172.21.15.63:0/2178743470 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f969c0b2a30 con 0x7f969c1430c0 2024-03-25T18:16:49.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 1 -- 172.21.15.63:0/2178743470 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1529432905 0 0) 0x7f9688003520 con 0x7f969c0a98f0 2024-03-25T18:16:49.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 1 -- 172.21.15.63:0/2178743470 >> v1:172.21.15.63:6790/0 conn(0x7f969c13f710 legacy=0x7f969c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:49.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 1 -- 172.21.15.63:0/2178743470 >> v1:172.21.15.63:6789/0 conn(0x7f969c1430c0 legacy=0x7f969c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:49.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:49.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 1 monclient: found mon.noname-c 2024-03-25T18:16:49.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:49.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:16:49.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 1 -- 172.21.15.63:0/2178743470 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c0b44c0 con 0x7f969c0a98f0 2024-03-25T18:16:49.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:49.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:49.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:49.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 1 -- 172.21.15.63:0/2178743470 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9698001690 con 0x7f969c0a98f0 2024-03-25T18:16:49.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f96a3972640 5 monclient: authenticate success, global_id 4417 2024-03-25T18:16:49.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f96a3972640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:49.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:49.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:49.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f9693fff640 10 monclient: dump: 2024-03-25T18:16:49.492 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:49.492 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:49.492 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:49.492 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:49.492 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:49.492 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:49.492 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:49.492 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:49.493 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:49.493 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:49.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.490+0000 7f96a3972640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:49.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.505+0000 7f9693fff640 1 -- 172.21.15.63:0/2178743470 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96980017f0 con 0x7f969c0a98f0 2024-03-25T18:16:49.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.505+0000 7f9693fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:49.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.505+0000 7f9693fff640 1 -- 172.21.15.63:0/2178743470 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9698001c00 con 0x7f969c0a98f0 2024-03-25T18:16:49.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.505+0000 7f9693fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:49.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.505+0000 7f9693fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:49.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.505+0000 7f9693fff640 10 monclient: dump: 2024-03-25T18:16:49.505 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:49.505 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:49.505 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:49.506 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:49.506 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:49.506 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:49.506 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:49.506 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:49.506 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:49.506 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:49.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.505+0000 7f96a3972640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:49.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.505+0000 7f96a3972640 4 set_mon_vals no callback set 2024-03-25T18:16:49.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.506+0000 7f96a3972640 10 monclient: shutdown 2024-03-25T18:16:49.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.506+0000 7f96a3972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:49.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.506+0000 7f96a3972640 1 -- 172.21.15.63:0/2178743470 >> v1:172.21.15.81:6789/0 conn(0x7f969c0a98f0 legacy=0x7f969c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:49.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.506+0000 7f96a3972640 1 -- 172.21.15.63:0/2178743470 shutdown_connections 2024-03-25T18:16:49.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.506+0000 7f96a3972640 1 -- 172.21.15.63:0/2178743470 >> 172.21.15.63:0/2178743470 conn(0x7f969c0b7ea0 msgr2=0x7f969c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:49.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.506+0000 7f96a3972640 1 -- 172.21.15.63:0/2178743470 shutdown_connections 2024-03-25T18:16:49.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.506+0000 7f96a3972640 1 -- 172.21.15.63:0/2178743470 wait complete. 2024-03-25T18:16:49.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.507+0000 7f96a3972640 10 monclient: build_initial_monmap 2024-03-25T18:16:49.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.507+0000 7f96a3972640 10 monclient: monmap: 2024-03-25T18:16:49.508 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:16:49.508 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:49.508 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:16:49.508 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:16:49.508 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:49.508 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:16:49.509 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:16:49.509 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:16:49.509 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:16:49.509 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:49.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.507+0000 7f96a3972640 1 librados: starting msgr at 2024-03-25T18:16:49.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.507+0000 7f96a3972640 1 librados: starting objecter 2024-03-25T18:16:49.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.508+0000 7f96a3972640 1 Processor -- start 2024-03-25T18:16:49.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.508+0000 7f96a3972640 1 -- start start 2024-03-25T18:16:49.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.508+0000 7f96a3972640 1 librados: setting wanted keys 2024-03-25T18:16:49.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.508+0000 7f96a3972640 1 librados: calling monclient init 2024-03-25T18:16:49.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.508+0000 7f96a3972640 10 monclient: init 2024-03-25T18:16:49.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.508+0000 7f96a3972640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:49.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.508+0000 7f96a3972640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:49.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.508+0000 7f96a3972640 10 monclient(hunting): picked mon.noname-c con 0x7f969c0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:16:49.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.508+0000 7f96a3972640 10 monclient(hunting): picked mon.noname-b con 0x7f969c13f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:16:49.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.508+0000 7f96a3972640 10 monclient(hunting): picked mon.noname-a con 0x7f969c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:16:49.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.508+0000 7f96a3972640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c138250 con 0x7f969c1430c0 2024-03-25T18:16:49.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.508+0000 7f96a3972640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c137d50 con 0x7f969c13f710 2024-03-25T18:16:49.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.508+0000 7f96a3972640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c137ee0 con 0x7f969c0a98f0 2024-03-25T18:16:49.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.508+0000 7f96a3972640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:49.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.508+0000 7f96a3972640 10 monclient(hunting): authenticate will time out at 7703.694168s 2024-03-25T18:16:49.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.525+0000 7f96a1f0f640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f969c1430c0 0x7f969c1d7df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59370/0 (socket says 172.21.15.63:59370) 2024-03-25T18:16:49.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.525+0000 7f96a1f0f640 1 -- 172.21.15.63:0/779233221 learned_addr learned my addr 172.21.15.63:0/779233221 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:16:49.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.527+0000 7f9691ffb640 10 client.?.objecter ms_handle_connect 0x7f969c1430c0 2024-03-25T18:16:49.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.527+0000 7f9691ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:49.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.530+0000 7f9691ffb640 10 client.?.objecter ms_handle_connect 0x7f969c13f710 2024-03-25T18:16:49.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.530+0000 7f9691ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:49.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.534+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899243784 0 0) 0x7f969c138250 con 0x7f969c1430c0 2024-03-25T18:16:49.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.534+0000 7f9691ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:49.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.534+0000 7f9691ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:49.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.534+0000 7f9691ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:49.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.534+0000 7f9691ffb640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:16:49.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.534+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9680004eb0 con 0x7f969c1430c0 2024-03-25T18:16:49.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.536+0000 7f9691ffb640 10 client.?.objecter ms_handle_connect 0x7f969c0a98f0 2024-03-25T18:16:49.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.536+0000 7f9691ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:49.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.541+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216187020 0 0) 0x7f969c137d50 con 0x7f969c13f710 2024-03-25T18:16:49.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.541+0000 7f9691ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:49.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.541+0000 7f9691ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:49.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.541+0000 7f9691ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:49.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.541+0000 7f9691ffb640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:16:49.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.541+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f969c138250 con 0x7f969c13f710 2024-03-25T18:16:49.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.548+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3449427975 0 0) 0x7f9680004eb0 con 0x7f969c1430c0 2024-03-25T18:16:49.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.548+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f969c137d50 con 0x7f969c1430c0 2024-03-25T18:16:49.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.548+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9694001c70 con 0x7f969c1430c0 2024-03-25T18:16:49.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.548+0000 7f9691ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:49.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.548+0000 7f9691ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:49.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.548+0000 7f9691ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:49.548 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:49.548 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:49.548 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:49.548 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:49.548 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:49.548 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:49.549 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:49.549 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:49.549 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:49.549 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:49.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.548+0000 7f9691ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:49.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.548+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261243497 0 0) 0x7f969c137ee0 con 0x7f969c0a98f0 2024-03-25T18:16:49.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.548+0000 7f9691ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:49.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.548+0000 7f9691ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:49.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.548+0000 7f9691ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:49.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.548+0000 7f9691ffb640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:16:49.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.548+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9680004eb0 con 0x7f969c0a98f0 2024-03-25T18:16:49.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.556+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2161780280 0 0) 0x7f969c138250 con 0x7f969c13f710 2024-03-25T18:16:49.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.557+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f969c137ee0 con 0x7f969c13f710 2024-03-25T18:16:49.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.557+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f968c001970 con 0x7f969c13f710 2024-03-25T18:16:49.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.557+0000 7f9691ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:49.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.557+0000 7f9691ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:49.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.557+0000 7f9691ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:49.557 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:49.557 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:49.557 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:49.557 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:49.557 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:49.557 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:49.558 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:49.558 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:49.558 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:49.558 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:49.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2655472171 0 0) 0x7f969c137d50 con 0x7f969c1430c0 2024-03-25T18:16:49.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 >> v1:172.21.15.63:6790/0 conn(0x7f969c0a98f0 legacy=0x7f969c137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:49.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 >> v1:172.21.15.81:6789/0 conn(0x7f969c13f710 legacy=0x7f969c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:49.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f9691ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:49.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f9691ffb640 1 monclient: found mon.a 2024-03-25T18:16:49.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f9691ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:49.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f9691ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:49.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c0b7730 con 0x7f969c1430c0 2024-03-25T18:16:49.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f9691ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:49.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f9691ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:49.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f9691ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:49.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f96a3972640 5 monclient: authenticate success, global_id 4401 2024-03-25T18:16:49.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f96a3972640 10 monclient: _renew_subs 2024-03-25T18:16:49.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f96a3972640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:49.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f96a3972640 1 -- 172.21.15.63:0/779233221 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969c0b7920 con 0x7f969c1430c0 2024-03-25T18:16:49.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f96a3972640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:49.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f96a3972640 10 monclient: _renew_subs 2024-03-25T18:16:49.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f96a3972640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:49.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f96a3972640 1 -- 172.21.15.63:0/779233221 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f969c1e33d0 con 0x7f969c1430c0 2024-03-25T18:16:49.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.560+0000 7f96a3972640 1 librados: init done 2024-03-25T18:16:49.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.561+0000 7f96a3972640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:49.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.561+0000 7f96a3972640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:49.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.561+0000 7f96a3972640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:49.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.561+0000 7f96a3972640 1 -- 172.21.15.63:0/779233221 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f969c0627b0 con 0x7f969c1430c0 2024-03-25T18:16:49.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.877+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9694003990 con 0x7f969c1430c0 2024-03-25T18:16:49.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.877+0000 7f9691ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:49.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.877+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9694003da0 con 0x7f969c1430c0 2024-03-25T18:16:49.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.877+0000 7f9691ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:49.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.877+0000 7f9691ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:49.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.877+0000 7f9691ffb640 10 monclient: dump: 2024-03-25T18:16:49.877 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:49.877 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:49.877 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:49.877 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:49.878 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:49.878 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:49.878 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:49.878 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:49.878 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:49.878 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:49.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.877+0000 7f9692ffd640 4 set_mon_vals no callback set 2024-03-25T18:16:49.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.878+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f9694003fc0 con 0x7f969c1430c0 2024-03-25T18:16:49.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.878+0000 7f96a170e640 10 monclient: get_auth_request con 0x7f968006dfb0 auth_method 0 2024-03-25T18:16:49.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.879+0000 7f9691ffb640 10 client.4401.objecter ms_handle_connect 0x7f968006dfb0 2024-03-25T18:16:49.885 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.884+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f9694054ae0 con 0x7f969c1430c0 2024-03-25T18:16:49.885 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.884+0000 7f9691ffb640 10 client.4401.objecter ms_dispatch 0x7f969c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:49.885 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.884+0000 7f9691ffb640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:49.885 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.884+0000 7f9691ffb640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:49.885 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.885+0000 7f9691ffb640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-25T18:16:49.885 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.885+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f96940505f0 con 0x7f969c1430c0 2024-03-25T18:16:49.885 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.885+0000 7f9691ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:49.885 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:49.885+0000 7f9691ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:50.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.099+0000 7f96a3972640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:50.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.099+0000 7f96a3972640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:50.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.099+0000 7f96a3972640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:16:50.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.099+0000 7f96a3972640 1 -- 172.21.15.63:0/779233221 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f969c1383e0 con 0x7f969c1430c0 2024-03-25T18:16:50.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.175+0000 7f9691ffb640 1 -- 172.21.15.63:0/779233221 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f9694052520 con 0x7f969c1430c0 2024-03-25T18:16:50.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.175+0000 7f9691ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:50.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.175+0000 7f9691ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:50.176 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:16:50.176 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:50.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.178+0000 7f96a3972640 10 librados: watch_flush enter 2024-03-25T18:16:50.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.178+0000 7f96a3972640 10 librados: watch_flush exit 2024-03-25T18:16:50.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.178+0000 7f96a3972640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-25T18:16:50.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.178+0000 7f96a3972640 10 client.4401.objecter successfully canceled tick 2024-03-25T18:16:50.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.178+0000 7f96a3972640 1 -- 172.21.15.63:0/779233221 >> v1:172.21.15.81:6800/3640340873 conn(0x7f968006dfb0 legacy=0x7f9680070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.178+0000 7f96a3972640 10 monclient: shutdown 2024-03-25T18:16:50.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.178+0000 7f96a3972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:50.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.178+0000 7f96a3972640 1 -- 172.21.15.63:0/779233221 >> v1:172.21.15.63:6789/0 conn(0x7f969c1430c0 legacy=0x7f969c1d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.178+0000 7f96a3972640 1 -- 172.21.15.63:0/779233221 shutdown_connections 2024-03-25T18:16:50.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.178+0000 7f96a3972640 1 -- 172.21.15.63:0/779233221 >> 172.21.15.63:0/779233221 conn(0x7f969c0b7ea0 msgr2=0x7f969c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:50.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.178+0000 7f96a3972640 1 -- 172.21.15.63:0/779233221 shutdown_connections 2024-03-25T18:16:50.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.178+0000 7f96a3972640 1 -- 172.21.15.63:0/779233221 wait complete. 2024-03-25T18:16:50.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:50.178+0000 7f96a3972640 1 librados: shutdown 2024-03-25T18:16:50.190 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:56.997 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:57.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.116+0000 7f48f1693640 10 monclient: get_monmap_and_config 2024-03-25T18:16:57.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.116+0000 7f48f1693640 10 monclient: build_initial_monmap 2024-03-25T18:16:57.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.116+0000 7f48f1693640 10 monclient: monmap: 2024-03-25T18:16:57.117 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:16:57.117 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:57.118 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:16:57.117635+0000 2024-03-25T18:16:57.118 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:16:57.117635+0000 2024-03-25T18:16:57.118 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:57.118 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:57.118 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:16:57.118 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:16:57.118 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:16:57.118 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:57.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.117+0000 7f48f1693640 1 Processor -- start 2024-03-25T18:16:57.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.117+0000 7f48f1693640 1 -- start start 2024-03-25T18:16:57.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.117+0000 7f48f1693640 10 monclient: init 2024-03-25T18:16:57.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.117+0000 7f48f1693640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:57.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.117+0000 7f48f1693640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:57.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.117+0000 7f48f1693640 10 monclient(hunting): picked mon.noname-b con 0x7f48ec0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:16:57.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.117+0000 7f48f1693640 10 monclient(hunting): picked mon.noname-a con 0x7f48ec13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:16:57.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.118+0000 7f48f1693640 10 monclient(hunting): picked mon.noname-c con 0x7f48ec1430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:16:57.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.118+0000 7f48f1693640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec142520 con 0x7f48ec13f710 2024-03-25T18:16:57.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.118+0000 7f48f1693640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec0b28a0 con 0x7f48ec1430c0 2024-03-25T18:16:57.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.118+0000 7f48f1693640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec0b2a30 con 0x7f48ec0a98f0 2024-03-25T18:16:57.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.118+0000 7f48f1693640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:57.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.118+0000 7f48f1693640 10 monclient(hunting): authenticate will time out at 7711.304434s 2024-03-25T18:16:57.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.120+0000 7f48eb7fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f48ec1430c0 0x7f48ec14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:58606/0 (socket says 172.21.15.63:58606) 2024-03-25T18:16:57.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.120+0000 7f48eb7fe640 1 -- 172.21.15.63:0/2167836382 learned_addr learned my addr 172.21.15.63:0/2167836382 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:16:57.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.129+0000 7f48e9ffb640 1 -- 172.21.15.63:0/2167836382 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970682563 0 0) 0x7f48ec0b28a0 con 0x7f48ec1430c0 2024-03-25T18:16:57.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.129+0000 7f48e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:57.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.129+0000 7f48e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:57.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.129+0000 7f48e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:57.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.129+0000 7f48e9ffb640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:16:57.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.129+0000 7f48e9ffb640 1 -- 172.21.15.63:0/2167836382 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48d4003520 con 0x7f48ec1430c0 2024-03-25T18:16:57.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.141+0000 7f48e9ffb640 1 -- 172.21.15.63:0/2167836382 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738723325 0 0) 0x7f48ec0b2a30 con 0x7f48ec0a98f0 2024-03-25T18:16:57.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.141+0000 7f48e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:57.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.141+0000 7f48e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:57.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.141+0000 7f48e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:57.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.141+0000 7f48e9ffb640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:16:57.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.141+0000 7f48e9ffb640 1 -- 172.21.15.63:0/2167836382 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ec0b28a0 con 0x7f48ec0a98f0 2024-03-25T18:16:57.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48e9ffb640 1 -- 172.21.15.63:0/2167836382 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1666318741 0 0) 0x7f48d4003520 con 0x7f48ec1430c0 2024-03-25T18:16:57.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48e9ffb640 1 -- 172.21.15.63:0/2167836382 >> v1:172.21.15.63:6790/0 conn(0x7f48ec0a98f0 legacy=0x7f48ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:57.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48e9ffb640 1 -- 172.21.15.63:0/2167836382 >> v1:172.21.15.63:6789/0 conn(0x7f48ec13f710 legacy=0x7f48ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:16:57.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48e9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:57.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48e9ffb640 1 monclient: found mon.noname-c 2024-03-25T18:16:57.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:57.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48e9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:16:57.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48e9ffb640 1 -- 172.21.15.63:0/2167836382 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec0b44c0 con 0x7f48ec1430c0 2024-03-25T18:16:57.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48e9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:57.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48e9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:57.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:57.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48e9ffb640 1 -- 172.21.15.63:0/2167836382 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f48e4001a10 con 0x7f48ec1430c0 2024-03-25T18:16:57.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:57.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48e9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:57.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48e9ffb640 10 monclient: dump: 2024-03-25T18:16:57.146 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:57.146 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:57.146 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:57.147 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:57.147 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:57.147 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:57.147 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:57.147 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:57.147 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:57.147 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:57.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48f1693640 5 monclient: authenticate success, global_id 4423 2024-03-25T18:16:57.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.143+0000 7f48f1693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:57.153 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.152+0000 7f48e9ffb640 1 -- 172.21.15.63:0/2167836382 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48e4001b70 con 0x7f48ec1430c0 2024-03-25T18:16:57.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.152+0000 7f48e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:57.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.152+0000 7f48e9ffb640 1 -- 172.21.15.63:0/2167836382 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f48e4001f80 con 0x7f48ec1430c0 2024-03-25T18:16:57.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.152+0000 7f48e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:57.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.152+0000 7f48e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:57.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.152+0000 7f48e9ffb640 10 monclient: dump: 2024-03-25T18:16:57.154 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:57.154 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:57.154 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:57.154 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:57.155 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:57.155 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:57.155 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:57.155 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:57.155 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:57.155 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:57.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.152+0000 7f48f1693640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:57.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.152+0000 7f48f1693640 4 set_mon_vals no callback set 2024-03-25T18:16:57.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.153+0000 7f48f1693640 10 monclient: shutdown 2024-03-25T18:16:57.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.153+0000 7f48f1693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:57.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.153+0000 7f48f1693640 1 -- 172.21.15.63:0/2167836382 >> v1:172.21.15.81:6789/0 conn(0x7f48ec1430c0 legacy=0x7f48ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:57.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.153+0000 7f48f1693640 1 -- 172.21.15.63:0/2167836382 shutdown_connections 2024-03-25T18:16:57.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.153+0000 7f48f1693640 1 -- 172.21.15.63:0/2167836382 >> 172.21.15.63:0/2167836382 conn(0x7f48ec0b7ea0 msgr2=0x7f48ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:57.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.153+0000 7f48f1693640 1 -- 172.21.15.63:0/2167836382 shutdown_connections 2024-03-25T18:16:57.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.153+0000 7f48f1693640 1 -- 172.21.15.63:0/2167836382 wait complete. 2024-03-25T18:16:57.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.154+0000 7f48f1693640 10 monclient: build_initial_monmap 2024-03-25T18:16:57.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.154+0000 7f48f1693640 10 monclient: monmap: 2024-03-25T18:16:57.158 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:16:57.158 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:57.159 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:16:57.159 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:16:57.159 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:57.159 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:16:57.159 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:16:57.159 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:16:57.159 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:16:57.159 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:57.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.155+0000 7f48f1693640 1 librados: starting msgr at 2024-03-25T18:16:57.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.155+0000 7f48f1693640 1 librados: starting objecter 2024-03-25T18:16:57.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.155+0000 7f48f1693640 1 Processor -- start 2024-03-25T18:16:57.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.155+0000 7f48f1693640 1 -- start start 2024-03-25T18:16:57.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.155+0000 7f48f1693640 1 librados: setting wanted keys 2024-03-25T18:16:57.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.155+0000 7f48f1693640 1 librados: calling monclient init 2024-03-25T18:16:57.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.155+0000 7f48f1693640 10 monclient: init 2024-03-25T18:16:57.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.156+0000 7f48f1693640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:57.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.156+0000 7f48f1693640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:57.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.156+0000 7f48f1693640 10 monclient(hunting): picked mon.noname-b con 0x7f48ec0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:16:57.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.156+0000 7f48f1693640 10 monclient(hunting): picked mon.noname-a con 0x7f48ec13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:16:57.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.156+0000 7f48f1693640 10 monclient(hunting): picked mon.noname-c con 0x7f48ec1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:16:57.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.156+0000 7f48f1693640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec1d7490 con 0x7f48ec13f710 2024-03-25T18:16:57.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.156+0000 7f48f1693640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec139980 con 0x7f48ec0a98f0 2024-03-25T18:16:57.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.156+0000 7f48f1693640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec1d8340 con 0x7f48ec1430c0 2024-03-25T18:16:57.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.156+0000 7f48f1693640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:57.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.156+0000 7f48f1693640 10 monclient(hunting): authenticate will time out at 7711.342722s 2024-03-25T18:16:57.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.157+0000 7f48eaffd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f48ec0a98f0 0x7f48ec138610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:58636/0 (socket says 172.21.15.63:58636) 2024-03-25T18:16:57.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.157+0000 7f48eaffd640 1 -- 172.21.15.63:0/3479802182 learned_addr learned my addr 172.21.15.63:0/3479802182 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:16:57.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.158+0000 7f48d37fe640 10 client.?.objecter ms_handle_connect 0x7f48ec0a98f0 2024-03-25T18:16:57.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.158+0000 7f48d37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:57.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.169+0000 7f48d37fe640 1 -- 172.21.15.63:0/3479802182 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598867004 0 0) 0x7f48ec139980 con 0x7f48ec0a98f0 2024-03-25T18:16:57.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.169+0000 7f48d37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:57.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.169+0000 7f48d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:57.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.169+0000 7f48d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:57.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.169+0000 7f48d37fe640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:16:57.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.169+0000 7f48d37fe640 1 -- 172.21.15.63:0/3479802182 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48d8004eb0 con 0x7f48ec0a98f0 2024-03-25T18:16:57.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.176+0000 7f48d37fe640 10 client.?.objecter ms_handle_connect 0x7f48ec1430c0 2024-03-25T18:16:57.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.176+0000 7f48d37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:57.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.176+0000 7f48d37fe640 1 -- 172.21.15.63:0/3479802182 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 11643899 0 0) 0x7f48d8004eb0 con 0x7f48ec0a98f0 2024-03-25T18:16:57.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.176+0000 7f48d37fe640 1 -- 172.21.15.63:0/3479802182 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48ec139980 con 0x7f48ec0a98f0 2024-03-25T18:16:57.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.176+0000 7f48d37fe640 1 -- 172.21.15.63:0/3479802182 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f48dc001a90 con 0x7f48ec0a98f0 2024-03-25T18:16:57.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.176+0000 7f48d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:57.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.176+0000 7f48d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:57.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.176+0000 7f48d37fe640 10 monclient(hunting): dump: 2024-03-25T18:16:57.178 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:57.178 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:57.178 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:57.178 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:57.179 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:57.179 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:57.179 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:57.179 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:57.179 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:57.179 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:57.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.176+0000 7f48d37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:57.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.185+0000 7f48d37fe640 1 -- 172.21.15.63:0/3479802182 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4254227080 0 0) 0x7f48ec139980 con 0x7f48ec0a98f0 2024-03-25T18:16:57.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.185+0000 7f48d37fe640 1 -- 172.21.15.63:0/3479802182 >> v1:172.21.15.63:6790/0 conn(0x7f48ec1430c0 legacy=0x7f48ec1e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:57.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.185+0000 7f48d37fe640 1 -- 172.21.15.63:0/3479802182 >> v1:172.21.15.63:6789/0 conn(0x7f48ec13f710 legacy=0x7f48ec1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:57.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.185+0000 7f48d37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:57.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.185+0000 7f48d37fe640 1 monclient: found mon.b 2024-03-25T18:16:57.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.185+0000 7f48d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:57.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.185+0000 7f48d37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:16:57.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.185+0000 7f48d37fe640 1 -- 172.21.15.63:0/3479802182 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec137870 con 0x7f48ec0a98f0 2024-03-25T18:16:57.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.185+0000 7f48d37fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:57.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.185+0000 7f48d37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:57.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.185+0000 7f48d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:57.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.185+0000 7f48f1693640 5 monclient: authenticate success, global_id 4426 2024-03-25T18:16:57.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.185+0000 7f48f1693640 10 monclient: _renew_subs 2024-03-25T18:16:57.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.185+0000 7f48f1693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:16:57.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.185+0000 7f48f1693640 1 -- 172.21.15.63:0/3479802182 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ec137b20 con 0x7f48ec0a98f0 2024-03-25T18:16:57.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.186+0000 7f48f1693640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:57.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.186+0000 7f48f1693640 10 monclient: _renew_subs 2024-03-25T18:16:57.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.186+0000 7f48f1693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:16:57.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.186+0000 7f48f1693640 1 -- 172.21.15.63:0/3479802182 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ec135220 con 0x7f48ec0a98f0 2024-03-25T18:16:57.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.186+0000 7f48f1693640 1 librados: init done 2024-03-25T18:16:57.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.187+0000 7f48f1693640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:57.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.187+0000 7f48f1693640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:57.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.187+0000 7f48f1693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:16:57.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.187+0000 7f48f1693640 1 -- 172.21.15.63:0/3479802182 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48ec0627b0 con 0x7f48ec0a98f0 2024-03-25T18:16:57.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.450+0000 7f48d37fe640 1 -- 172.21.15.63:0/3479802182 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48dc0038c0 con 0x7f48ec0a98f0 2024-03-25T18:16:57.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.450+0000 7f48d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:57.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.450+0000 7f48d37fe640 1 -- 172.21.15.63:0/3479802182 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f48dc003c90 con 0x7f48ec0a98f0 2024-03-25T18:16:57.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.450+0000 7f48d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:57.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.450+0000 7f48d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:57.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.450+0000 7f48d37fe640 10 monclient: dump: 2024-03-25T18:16:57.452 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:16:57.452 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:16:57.452 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:57.452 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:16:57.452 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:57.452 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:16:57.453 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:16:57.453 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:16:57.453 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:16:57.453 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:16:57.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.450+0000 7f48e97fa640 4 set_mon_vals no callback set 2024-03-25T18:16:57.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.451+0000 7f48d37fe640 1 -- 172.21.15.63:0/3479802182 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f48dc003eb0 con 0x7f48ec0a98f0 2024-03-25T18:16:57.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.452+0000 7f48ea7fc640 10 monclient: get_auth_request con 0x7f48d806dc70 auth_method 0 2024-03-25T18:16:57.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.452+0000 7f48d37fe640 10 client.4426.objecter ms_handle_connect 0x7f48d806dc70 2024-03-25T18:16:57.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.455+0000 7f48d37fe640 1 -- 172.21.15.63:0/3479802182 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f48dc054a10 con 0x7f48ec0a98f0 2024-03-25T18:16:57.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.455+0000 7f48d37fe640 10 client.4426.objecter ms_dispatch 0x7f48ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:57.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.455+0000 7f48d37fe640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:57.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.455+0000 7f48d37fe640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:57.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.456+0000 7f48d37fe640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-25T18:16:57.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.456+0000 7f48d37fe640 1 -- 172.21.15.63:0/3479802182 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f48dc050520 con 0x7f48ec0a98f0 2024-03-25T18:16:57.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.456+0000 7f48d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:57.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.456+0000 7f48d37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:57.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.689+0000 7f48f1693640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:57.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.689+0000 7f48f1693640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:57.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.689+0000 7f48f1693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:16:57.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.689+0000 7f48f1693640 1 -- 172.21.15.63:0/3479802182 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48ec1d7aa0 con 0x7f48ec0a98f0 2024-03-25T18:16:57.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.723+0000 7f48d37fe640 1 -- 172.21.15.63:0/3479802182 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f48dc052450 con 0x7f48ec0a98f0 2024-03-25T18:16:57.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.723+0000 7f48d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:57.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.723+0000 7f48d37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:57.724 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:16:57.725 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:57.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.725+0000 7f48f1693640 10 librados: watch_flush enter 2024-03-25T18:16:57.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.725+0000 7f48f1693640 10 librados: watch_flush exit 2024-03-25T18:16:57.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.725+0000 7f48f1693640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-25T18:16:57.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.725+0000 7f48f1693640 10 client.4426.objecter successfully canceled tick 2024-03-25T18:16:57.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.725+0000 7f48f1693640 1 -- 172.21.15.63:0/3479802182 >> v1:172.21.15.81:6800/3640340873 conn(0x7f48d806dc70 legacy=0x7f48d8070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:57.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.725+0000 7f48f1693640 10 monclient: shutdown 2024-03-25T18:16:57.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.725+0000 7f48f1693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:57.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.725+0000 7f48f1693640 1 -- 172.21.15.63:0/3479802182 >> v1:172.21.15.81:6789/0 conn(0x7f48ec0a98f0 legacy=0x7f48ec138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:57.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.726+0000 7f48f1693640 1 -- 172.21.15.63:0/3479802182 shutdown_connections 2024-03-25T18:16:57.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.726+0000 7f48f1693640 1 -- 172.21.15.63:0/3479802182 >> 172.21.15.63:0/3479802182 conn(0x7f48ec0b7ea0 msgr2=0x7f48ec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:57.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.726+0000 7f48f1693640 1 -- 172.21.15.63:0/3479802182 shutdown_connections 2024-03-25T18:16:57.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.726+0000 7f48f1693640 1 -- 172.21.15.63:0/3479802182 wait complete. 2024-03-25T18:16:57.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:16:57.726+0000 7f48f1693640 1 librados: shutdown 2024-03-25T18:16:57.738 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:58.846 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:04:40.539 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:58.852 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:04:40.544 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:58.896 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:04:40.557 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:58.902 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:04:40.594 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:58.908 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:04:40.569 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:58.911 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:04:40.603 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:58.920 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:04:40.612 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:58.939 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:04:40.600 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:58.953 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:04:40.646 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:04.544 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:04.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.658+0000 7f813059d640 10 monclient: get_monmap_and_config 2024-03-25T18:17:04.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.658+0000 7f813059d640 10 monclient: build_initial_monmap 2024-03-25T18:17:04.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.658+0000 7f813059d640 10 monclient: monmap: 2024-03-25T18:17:04.660 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:17:04.660 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:04.660 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:17:04.659378+0000 2024-03-25T18:17:04.660 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:17:04.659378+0000 2024-03-25T18:17:04.660 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:04.660 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:04.660 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:17:04.660 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:17:04.660 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:17:04.661 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:04.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.659+0000 7f813059d640 1 Processor -- start 2024-03-25T18:17:04.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.659+0000 7f813059d640 1 -- start start 2024-03-25T18:17:04.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.659+0000 7f813059d640 10 monclient: init 2024-03-25T18:17:04.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.659+0000 7f813059d640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:04.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.659+0000 7f813059d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:04.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.660+0000 7f813059d640 10 monclient(hunting): picked mon.noname-a con 0x7f81280a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:17:04.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.660+0000 7f813059d640 10 monclient(hunting): picked mon.noname-c con 0x7f812813f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:17:04.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.660+0000 7f813059d640 10 monclient(hunting): picked mon.noname-b con 0x7f81281430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:17:04.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.660+0000 7f813059d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8128142520 con 0x7f81280a98f0 2024-03-25T18:17:04.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.660+0000 7f813059d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81280b28a0 con 0x7f812813f710 2024-03-25T18:17:04.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.660+0000 7f813059d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81280b2a30 con 0x7f81281430c0 2024-03-25T18:17:04.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.660+0000 7f813059d640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:04.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.660+0000 7f813059d640 10 monclient(hunting): authenticate will time out at 7718.846514s 2024-03-25T18:17:04.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.665+0000 7f812e339640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f81280a98f0 0x7f81280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:44358/0 (socket says 172.21.15.63:44358) 2024-03-25T18:17:04.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.665+0000 7f812e339640 1 -- 172.21.15.63:0/2121846383 learned_addr learned my addr 172.21.15.63:0/2121846383 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:17:04.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.683+0000 7f812d337640 1 -- 172.21.15.63:0/2121846383 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849416076 0 0) 0x7f8128142520 con 0x7f81280a98f0 2024-03-25T18:17:04.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.683+0000 7f812d337640 10 monclient(hunting): _negotiate 2024-03-25T18:17:04.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.683+0000 7f812d337640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:04.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.683+0000 7f812d337640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:04.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.683+0000 7f812d337640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:17:04.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.683+0000 7f812d337640 1 -- 172.21.15.63:0/2121846383 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8114003520 con 0x7f81280a98f0 2024-03-25T18:17:04.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.691+0000 7f812d337640 1 -- 172.21.15.63:0/2121846383 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4014130502 0 0) 0x7f8114003520 con 0x7f81280a98f0 2024-03-25T18:17:04.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.691+0000 7f812d337640 1 -- 172.21.15.63:0/2121846383 >> v1:172.21.15.63:6790/0 conn(0x7f81281430c0 legacy=0x7f812814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:04.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.691+0000 7f812d337640 1 -- 172.21.15.63:0/2121846383 >> v1:172.21.15.81:6789/0 conn(0x7f812813f710 legacy=0x7f81281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:04.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.691+0000 7f812d337640 10 monclient: _finish_hunting 0 2024-03-25T18:17:04.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.691+0000 7f812d337640 1 monclient: found mon.noname-a 2024-03-25T18:17:04.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.691+0000 7f812d337640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:04.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.691+0000 7f812d337640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:17:04.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.691+0000 7f812d337640 1 -- 172.21.15.63:0/2121846383 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81280b44c0 con 0x7f81280a98f0 2024-03-25T18:17:04.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.692+0000 7f812d337640 10 monclient: _finish_auth 0 2024-03-25T18:17:04.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.692+0000 7f812d337640 10 monclient: _check_auth_tickets 2024-03-25T18:17:04.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.692+0000 7f812d337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:04.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.692+0000 7f812d337640 1 -- 172.21.15.63:0/2121846383 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8124001690 con 0x7f81280a98f0 2024-03-25T18:17:04.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.692+0000 7f812d337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:04.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.692+0000 7f812d337640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:04.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.692+0000 7f812d337640 10 monclient: dump: 2024-03-25T18:17:04.694 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:04.694 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:04.694 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:04.694 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:04.694 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:04.694 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:04.694 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:04.694 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:04.694 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:04.694 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:04.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.692+0000 7f813059d640 5 monclient: authenticate success, global_id 4404 2024-03-25T18:17:04.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.692+0000 7f813059d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:04.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.697+0000 7f812d337640 1 -- 172.21.15.63:0/2121846383 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81240017f0 con 0x7f81280a98f0 2024-03-25T18:17:04.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.697+0000 7f812d337640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:04.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.697+0000 7f812d337640 1 -- 172.21.15.63:0/2121846383 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8124001c00 con 0x7f81280a98f0 2024-03-25T18:17:04.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.697+0000 7f813059d640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:04.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.697+0000 7f813059d640 4 set_mon_vals no callback set 2024-03-25T18:17:04.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.697+0000 7f812d337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:04.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.697+0000 7f812d337640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:04.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.697+0000 7f812d337640 10 monclient: dump: 2024-03-25T18:17:04.699 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:04.699 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:04.699 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:04.699 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:04.699 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:04.699 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:04.699 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:04.699 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:04.699 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:04.699 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:04.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.698+0000 7f813059d640 10 monclient: shutdown 2024-03-25T18:17:04.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.698+0000 7f813059d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:04.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.698+0000 7f813059d640 1 -- 172.21.15.63:0/2121846383 >> v1:172.21.15.63:6789/0 conn(0x7f81280a98f0 legacy=0x7f81280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:04.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.698+0000 7f813059d640 1 -- 172.21.15.63:0/2121846383 shutdown_connections 2024-03-25T18:17:04.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.698+0000 7f813059d640 1 -- 172.21.15.63:0/2121846383 >> 172.21.15.63:0/2121846383 conn(0x7f81280b7ea0 msgr2=0x7f81280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:04.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.698+0000 7f813059d640 1 -- 172.21.15.63:0/2121846383 shutdown_connections 2024-03-25T18:17:04.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.698+0000 7f813059d640 1 -- 172.21.15.63:0/2121846383 wait complete. 2024-03-25T18:17:04.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.699+0000 7f813059d640 10 monclient: build_initial_monmap 2024-03-25T18:17:04.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.699+0000 7f813059d640 10 monclient: monmap: 2024-03-25T18:17:04.701 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:17:04.701 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:04.701 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:17:04.701 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:17:04.701 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:04.701 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:17:04.701 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:17:04.702 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:17:04.702 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:17:04.702 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:04.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.699+0000 7f813059d640 1 librados: starting msgr at 2024-03-25T18:17:04.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.699+0000 7f813059d640 1 librados: starting objecter 2024-03-25T18:17:04.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.699+0000 7f813059d640 1 Processor -- start 2024-03-25T18:17:04.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.699+0000 7f813059d640 1 -- start start 2024-03-25T18:17:04.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.699+0000 7f813059d640 1 librados: setting wanted keys 2024-03-25T18:17:04.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.699+0000 7f813059d640 1 librados: calling monclient init 2024-03-25T18:17:04.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.699+0000 7f813059d640 10 monclient: init 2024-03-25T18:17:04.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.700+0000 7f813059d640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:04.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.700+0000 7f813059d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:04.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.700+0000 7f813059d640 10 monclient(hunting): picked mon.noname-a con 0x7f81280a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:17:04.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.700+0000 7f813059d640 10 monclient(hunting): picked mon.noname-b con 0x7f812813f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:17:04.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.700+0000 7f813059d640 10 monclient(hunting): picked mon.noname-c con 0x7f81281430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:17:04.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.700+0000 7f813059d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81280abd30 con 0x7f81280a98f0 2024-03-25T18:17:04.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.700+0000 7f813059d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81281d77a0 con 0x7f812813f710 2024-03-25T18:17:04.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.700+0000 7f813059d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81281d51f0 con 0x7f81281430c0 2024-03-25T18:17:04.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.700+0000 7f813059d640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:04.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.700+0000 7f813059d640 10 monclient(hunting): authenticate will time out at 7718.886921s 2024-03-25T18:17:04.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.719+0000 7f812db38640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f812813f710 0x7f8128138340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:41502/0 (socket says 172.21.15.63:41502) 2024-03-25T18:17:04.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.719+0000 7f812db38640 1 -- 172.21.15.63:0/685106671 learned_addr learned my addr 172.21.15.63:0/685106671 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:17:04.721 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.720+0000 7f811effd640 10 client.?.objecter ms_handle_connect 0x7f81280a98f0 2024-03-25T18:17:04.721 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.720+0000 7f811effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:04.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.722+0000 7f811effd640 10 client.?.objecter ms_handle_connect 0x7f81281430c0 2024-03-25T18:17:04.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.722+0000 7f811effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:04.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.722+0000 7f811effd640 10 client.?.objecter ms_handle_connect 0x7f812813f710 2024-03-25T18:17:04.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.722+0000 7f811effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:04.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.729+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082003066 0 0) 0x7f81281d77a0 con 0x7f812813f710 2024-03-25T18:17:04.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.729+0000 7f811effd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:04.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.729+0000 7f811effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:04.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.729+0000 7f811effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:04.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.729+0000 7f811effd640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:17:04.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.729+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f810c004eb0 con 0x7f812813f710 2024-03-25T18:17:04.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.732+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125475243 0 0) 0x7f81280abd30 con 0x7f81280a98f0 2024-03-25T18:17:04.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.732+0000 7f811effd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:04.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.732+0000 7f811effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:04.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.732+0000 7f811effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:04.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.732+0000 7f811effd640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:17:04.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.732+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81281d77a0 con 0x7f81280a98f0 2024-03-25T18:17:04.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.743+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2140689822 0 0) 0x7f810c004eb0 con 0x7f812813f710 2024-03-25T18:17:04.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.743+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f810c005490 con 0x7f812813f710 2024-03-25T18:17:04.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.743+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8118001a30 con 0x7f812813f710 2024-03-25T18:17:04.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.743+0000 7f811effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:04.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.743+0000 7f811effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:04.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.743+0000 7f811effd640 10 monclient(hunting): dump: 2024-03-25T18:17:04.744 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:04.744 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:04.744 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:04.744 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:04.744 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:04.745 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:04.745 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:04.745 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:04.745 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:04.745 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:04.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.743+0000 7f811effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:04.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.743+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560212590 0 0) 0x7f81281d51f0 con 0x7f81281430c0 2024-03-25T18:17:04.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.743+0000 7f811effd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:04.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.743+0000 7f811effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:04.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.743+0000 7f811effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:04.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.743+0000 7f811effd640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:17:04.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.743+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f810c004eb0 con 0x7f81281430c0 2024-03-25T18:17:04.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.744+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1850787384 0 0) 0x7f81281d77a0 con 0x7f81280a98f0 2024-03-25T18:17:04.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.744+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81281d51f0 con 0x7f81280a98f0 2024-03-25T18:17:04.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.744+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8124001b00 con 0x7f81280a98f0 2024-03-25T18:17:04.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.744+0000 7f811effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:04.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.744+0000 7f811effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:04.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.744+0000 7f811effd640 10 monclient(hunting): dump: 2024-03-25T18:17:04.746 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:04.746 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:04.747 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:04.747 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:04.747 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:04.747 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:04.747 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:04.747 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:04.747 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:04.747 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:04.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.749+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1104240139 0 0) 0x7f81281d51f0 con 0x7f81280a98f0 2024-03-25T18:17:04.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 >> v1:172.21.15.63:6790/0 conn(0x7f81281430c0 legacy=0x7f81281e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:04.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 >> v1:172.21.15.81:6789/0 conn(0x7f812813f710 legacy=0x7f8128138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:04.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f811effd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:04.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f811effd640 1 monclient: found mon.a 2024-03-25T18:17:04.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f811effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:04.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f811effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:17:04.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81281d83d0 con 0x7f81280a98f0 2024-03-25T18:17:04.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f811effd640 10 monclient: _finish_auth 0 2024-03-25T18:17:04.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f811effd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:04.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f811effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:04.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f813059d640 5 monclient: authenticate success, global_id 4407 2024-03-25T18:17:04.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f813059d640 10 monclient: _renew_subs 2024-03-25T18:17:04.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f813059d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:17:04.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f813059d640 1 -- 172.21.15.63:0/685106671 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8128138a50 con 0x7f81280a98f0 2024-03-25T18:17:04.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f813059d640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:04.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f813059d640 10 monclient: _renew_subs 2024-03-25T18:17:04.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f813059d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:17:04.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f813059d640 1 -- 172.21.15.63:0/685106671 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81281e35c0 con 0x7f81280a98f0 2024-03-25T18:17:04.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.750+0000 7f813059d640 1 librados: init done 2024-03-25T18:17:04.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.751+0000 7f813059d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:04.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.751+0000 7f813059d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:04.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.751+0000 7f813059d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:17:04.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:04.751+0000 7f813059d640 1 -- 172.21.15.63:0/685106671 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81280627b0 con 0x7f81280a98f0 2024-03-25T18:17:05.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.068+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8124001c60 con 0x7f81280a98f0 2024-03-25T18:17:05.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.068+0000 7f811effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:05.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.068+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8124004c30 con 0x7f81280a98f0 2024-03-25T18:17:05.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.068+0000 7f811effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:05.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.068+0000 7f811effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:05.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.068+0000 7f811effd640 10 monclient: dump: 2024-03-25T18:17:05.070 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:05.070 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:05.070 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:05.070 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:05.070 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:05.070 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:05.070 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:05.071 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:05.071 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:05.071 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:05.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.068+0000 7f812cb36640 4 set_mon_vals no callback set 2024-03-25T18:17:05.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.069+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f812401ac00 con 0x7f81280a98f0 2024-03-25T18:17:05.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.070+0000 7f812db38640 10 monclient: get_auth_request con 0x7f810c06e100 auth_method 0 2024-03-25T18:17:05.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.070+0000 7f811effd640 10 client.4407.objecter ms_handle_connect 0x7f810c06e100 2024-03-25T18:17:05.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.077+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f8124054730 con 0x7f81280a98f0 2024-03-25T18:17:05.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.077+0000 7f811effd640 10 client.4407.objecter ms_dispatch 0x7f8128000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:05.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.077+0000 7f811effd640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:05.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.077+0000 7f811effd640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:05.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.077+0000 7f811effd640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-25T18:17:05.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.077+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8124050070 con 0x7f81280a98f0 2024-03-25T18:17:05.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.077+0000 7f811effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:05.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.077+0000 7f811effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:05.337 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.336+0000 7f813059d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:05.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.336+0000 7f813059d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:05.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.336+0000 7f813059d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:17:05.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.336+0000 7f813059d640 1 -- 172.21.15.63:0/685106671 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8128074cc0 con 0x7f81280a98f0 2024-03-25T18:17:05.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.359+0000 7f811effd640 1 -- 172.21.15.63:0/685106671 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f8124052170 con 0x7f81280a98f0 2024-03-25T18:17:05.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.359+0000 7f811effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:05.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.359+0000 7f811effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:05.360 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:17:05.361 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:05.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.362+0000 7f813059d640 10 librados: watch_flush enter 2024-03-25T18:17:05.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.362+0000 7f813059d640 10 librados: watch_flush exit 2024-03-25T18:17:05.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.362+0000 7f813059d640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-25T18:17:05.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.362+0000 7f813059d640 10 client.4407.objecter successfully canceled tick 2024-03-25T18:17:05.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.362+0000 7f813059d640 1 -- 172.21.15.63:0/685106671 >> v1:172.21.15.81:6800/3640340873 conn(0x7f810c06e100 legacy=0x7f810c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:05.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.362+0000 7f813059d640 10 monclient: shutdown 2024-03-25T18:17:05.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.362+0000 7f813059d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:05.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.362+0000 7f813059d640 1 -- 172.21.15.63:0/685106671 >> v1:172.21.15.63:6789/0 conn(0x7f81280a98f0 legacy=0x7f81281d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:05.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.362+0000 7f813059d640 1 -- 172.21.15.63:0/685106671 shutdown_connections 2024-03-25T18:17:05.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.362+0000 7f813059d640 1 -- 172.21.15.63:0/685106671 >> 172.21.15.63:0/685106671 conn(0x7f81280b7ea0 msgr2=0x7f8128074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:05.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.362+0000 7f813059d640 1 -- 172.21.15.63:0/685106671 shutdown_connections 2024-03-25T18:17:05.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.362+0000 7f813059d640 1 -- 172.21.15.63:0/685106671 wait complete. 2024-03-25T18:17:05.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:05.362+0000 7f813059d640 1 librados: shutdown 2024-03-25T18:17:05.375 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:12.181 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:12.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.294+0000 7f68734d9640 10 monclient: get_monmap_and_config 2024-03-25T18:17:12.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.294+0000 7f68734d9640 10 monclient: build_initial_monmap 2024-03-25T18:17:12.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.294+0000 7f68734d9640 10 monclient: monmap: 2024-03-25T18:17:12.295 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:17:12.295 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:12.295 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:17:12.295413+0000 2024-03-25T18:17:12.295 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:17:12.295413+0000 2024-03-25T18:17:12.295 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:12.296 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:12.296 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:17:12.296 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:17:12.296 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:17:12.296 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:12.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.294+0000 7f68734d9640 1 Processor -- start 2024-03-25T18:17:12.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.295+0000 7f68734d9640 1 -- start start 2024-03-25T18:17:12.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.295+0000 7f68734d9640 10 monclient: init 2024-03-25T18:17:12.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.295+0000 7f68734d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:12.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.295+0000 7f68734d9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:12.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.295+0000 7f68734d9640 10 monclient(hunting): picked mon.noname-b con 0x7f686c0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:17:12.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.295+0000 7f68734d9640 10 monclient(hunting): picked mon.noname-c con 0x7f686c13f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:17:12.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.295+0000 7f68734d9640 10 monclient(hunting): picked mon.noname-a con 0x7f686c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:17:12.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.295+0000 7f68734d9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c142520 con 0x7f686c1430c0 2024-03-25T18:17:12.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.296+0000 7f68734d9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c0b28a0 con 0x7f686c13f710 2024-03-25T18:17:12.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.296+0000 7f68734d9640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c0b2a30 con 0x7f686c0a98f0 2024-03-25T18:17:12.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.296+0000 7f68734d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:12.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.296+0000 7f68734d9640 10 monclient(hunting): authenticate will time out at 7726.482527s 2024-03-25T18:17:12.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.296+0000 7f6870a74640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f686c13f710 0x7f686c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:57048/0 (socket says 172.21.15.63:57048) 2024-03-25T18:17:12.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.296+0000 7f6870a74640 1 -- 172.21.15.63:0/3467663000 learned_addr learned my addr 172.21.15.63:0/3467663000 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:17:12.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.373+0000 7f6863fff640 1 -- 172.21.15.63:0/3467663000 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309504101 0 0) 0x7f686c0b2a30 con 0x7f686c0a98f0 2024-03-25T18:17:12.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.373+0000 7f6863fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:12.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.373+0000 7f6863fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:12.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.373+0000 7f6863fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:12.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.373+0000 7f6863fff640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:17:12.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.373+0000 7f6863fff640 1 -- 172.21.15.63:0/3467663000 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6858003520 con 0x7f686c0a98f0 2024-03-25T18:17:12.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.388+0000 7f6863fff640 1 -- 172.21.15.63:0/3467663000 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298676855 0 0) 0x7f686c142520 con 0x7f686c1430c0 2024-03-25T18:17:12.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.388+0000 7f6863fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:12.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.388+0000 7f6863fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:12.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.388+0000 7f6863fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:12.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.388+0000 7f6863fff640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:17:12.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.388+0000 7f6863fff640 1 -- 172.21.15.63:0/3467663000 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f686c0b2a30 con 0x7f686c1430c0 2024-03-25T18:17:12.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.394+0000 7f6863fff640 1 -- 172.21.15.63:0/3467663000 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752318692 0 0) 0x7f686c0b28a0 con 0x7f686c13f710 2024-03-25T18:17:12.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.394+0000 7f6863fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:12.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.394+0000 7f6863fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:12.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.394+0000 7f6863fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:12.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.394+0000 7f6863fff640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:17:12.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.394+0000 7f6863fff640 1 -- 172.21.15.63:0/3467663000 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f686c142520 con 0x7f686c13f710 2024-03-25T18:17:12.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f6863fff640 1 -- 172.21.15.63:0/3467663000 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3845033000 0 0) 0x7f686c0b2a30 con 0x7f686c1430c0 2024-03-25T18:17:12.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f6863fff640 1 -- 172.21.15.63:0/3467663000 >> v1:172.21.15.63:6790/0 conn(0x7f686c0a98f0 legacy=0x7f686c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:12.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f6863fff640 1 -- 172.21.15.63:0/3467663000 >> v1:172.21.15.81:6789/0 conn(0x7f686c13f710 legacy=0x7f686c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:12.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f6863fff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:12.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f6863fff640 1 monclient: found mon.noname-a 2024-03-25T18:17:12.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f6863fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:12.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f6863fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:17:12.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f6863fff640 1 -- 172.21.15.63:0/3467663000 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686c0b44c0 con 0x7f686c1430c0 2024-03-25T18:17:12.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f6863fff640 10 monclient: _finish_auth 0 2024-03-25T18:17:12.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f6863fff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:12.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f6863fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:12.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f6863fff640 1 -- 172.21.15.63:0/3467663000 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f68640019f0 con 0x7f686c1430c0 2024-03-25T18:17:12.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f68734d9640 5 monclient: authenticate success, global_id 4410 2024-03-25T18:17:12.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f68734d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:12.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f6863fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:12.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f6863fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:12.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.403+0000 7f6863fff640 10 monclient: dump: 2024-03-25T18:17:12.406 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:12.406 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:12.406 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:12.406 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:12.406 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:12.406 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:12.406 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:12.407 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:12.407 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:12.407 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:12.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.404+0000 7f68734d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.411+0000 7f6863fff640 1 -- 172.21.15.63:0/3467663000 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6864001b50 con 0x7f686c1430c0 2024-03-25T18:17:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.411+0000 7f6863fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.411+0000 7f6863fff640 1 -- 172.21.15.63:0/3467663000 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6864001f60 con 0x7f686c1430c0 2024-03-25T18:17:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.411+0000 7f6863fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.411+0000 7f6863fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.411+0000 7f6863fff640 10 monclient: dump: 2024-03-25T18:17:12.412 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:12.413 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:12.413 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:12.413 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:12.413 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:12.413 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:12.413 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:12.413 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:12.413 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:12.413 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:12.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.411+0000 7f68734d9640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:12.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.411+0000 7f68734d9640 4 set_mon_vals no callback set 2024-03-25T18:17:12.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.412+0000 7f68734d9640 10 monclient: shutdown 2024-03-25T18:17:12.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.412+0000 7f68734d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:12.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.412+0000 7f68734d9640 1 -- 172.21.15.63:0/3467663000 >> v1:172.21.15.63:6789/0 conn(0x7f686c1430c0 legacy=0x7f686c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:12.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.412+0000 7f68734d9640 1 -- 172.21.15.63:0/3467663000 shutdown_connections 2024-03-25T18:17:12.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.412+0000 7f68734d9640 1 -- 172.21.15.63:0/3467663000 >> 172.21.15.63:0/3467663000 conn(0x7f686c0b7ea0 msgr2=0x7f686c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:12.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.412+0000 7f68734d9640 1 -- 172.21.15.63:0/3467663000 shutdown_connections 2024-03-25T18:17:12.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.413+0000 7f68734d9640 1 -- 172.21.15.63:0/3467663000 wait complete. 2024-03-25T18:17:12.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.414+0000 7f68734d9640 10 monclient: build_initial_monmap 2024-03-25T18:17:12.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.414+0000 7f68734d9640 10 monclient: monmap: 2024-03-25T18:17:12.416 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:17:12.416 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:12.416 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:17:12.417 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:17:12.417 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:12.417 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:17:12.417 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:17:12.417 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:17:12.417 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:17:12.417 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:12.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.414+0000 7f68734d9640 1 librados: starting msgr at 2024-03-25T18:17:12.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.414+0000 7f68734d9640 1 librados: starting objecter 2024-03-25T18:17:12.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.414+0000 7f68734d9640 1 Processor -- start 2024-03-25T18:17:12.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.414+0000 7f68734d9640 1 -- start start 2024-03-25T18:17:12.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.414+0000 7f68734d9640 1 librados: setting wanted keys 2024-03-25T18:17:12.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.414+0000 7f68734d9640 1 librados: calling monclient init 2024-03-25T18:17:12.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.414+0000 7f68734d9640 10 monclient: init 2024-03-25T18:17:12.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.415+0000 7f68734d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:12.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.415+0000 7f68734d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:12.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.415+0000 7f68734d9640 10 monclient(hunting): picked mon.noname-b con 0x7f686c0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:17:12.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.415+0000 7f68734d9640 10 monclient(hunting): picked mon.noname-c con 0x7f686c13f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:17:12.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.415+0000 7f68734d9640 10 monclient(hunting): picked mon.noname-a con 0x7f686c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:17:12.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.415+0000 7f68734d9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c1d48f0 con 0x7f686c1430c0 2024-03-25T18:17:12.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.415+0000 7f68734d9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c1397d0 con 0x7f686c0a98f0 2024-03-25T18:17:12.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.415+0000 7f68734d9640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c1d4b50 con 0x7f686c13f710 2024-03-25T18:17:12.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.415+0000 7f68734d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:12.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.415+0000 7f68734d9640 10 monclient(hunting): authenticate will time out at 7726.601724s 2024-03-25T18:17:12.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.439+0000 7f6871275640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f686c0a98f0 0x7f686c1d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:57124/0 (socket says 172.21.15.63:57124) 2024-03-25T18:17:12.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.439+0000 7f6871275640 1 -- 172.21.15.63:0/2912791461 learned_addr learned my addr 172.21.15.63:0/2912791461 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:17:12.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.446+0000 7f6861ffb640 10 client.?.objecter ms_handle_connect 0x7f686c0a98f0 2024-03-25T18:17:12.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.446+0000 7f6861ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:12.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.448+0000 7f6861ffb640 10 client.?.objecter ms_handle_connect 0x7f686c1430c0 2024-03-25T18:17:12.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.448+0000 7f6861ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:12.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.457+0000 7f6861ffb640 10 client.?.objecter ms_handle_connect 0x7f686c13f710 2024-03-25T18:17:12.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.457+0000 7f6861ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:12.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.458+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841124562 0 0) 0x7f686c1d48f0 con 0x7f686c1430c0 2024-03-25T18:17:12.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.458+0000 7f6861ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:12.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.458+0000 7f6861ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:12.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.458+0000 7f6861ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:12.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.458+0000 7f6861ffb640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:17:12.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.458+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6850004eb0 con 0x7f686c1430c0 2024-03-25T18:17:12.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.465+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958100790 0 0) 0x7f686c1d4b50 con 0x7f686c13f710 2024-03-25T18:17:12.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.465+0000 7f6861ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:12.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.465+0000 7f6861ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:12.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.465+0000 7f6861ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:12.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.465+0000 7f6861ffb640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:17:12.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.465+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f686c1d48f0 con 0x7f686c13f710 2024-03-25T18:17:12.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.467+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378406611 0 0) 0x7f686c1397d0 con 0x7f686c0a98f0 2024-03-25T18:17:12.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.467+0000 7f6861ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:12.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.467+0000 7f6861ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:12.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.467+0000 7f6861ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:12.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.467+0000 7f6861ffb640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:17:12.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.467+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f686c1d4b50 con 0x7f686c0a98f0 2024-03-25T18:17:12.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.475+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3336002297 0 0) 0x7f6850004eb0 con 0x7f686c1430c0 2024-03-25T18:17:12.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.475+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f686c1397d0 con 0x7f686c1430c0 2024-03-25T18:17:12.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.475+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6864001ea0 con 0x7f686c1430c0 2024-03-25T18:17:12.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.475+0000 7f6861ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:12.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.475+0000 7f6861ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:12.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.475+0000 7f6861ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:12.476 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:12.477 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:12.477 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:12.477 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:12.477 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:12.477 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:12.477 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:12.477 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:12.477 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:12.477 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:12.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.475+0000 7f6861ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3288543130 0 0) 0x7f686c1397d0 con 0x7f686c1430c0 2024-03-25T18:17:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 >> v1:172.21.15.63:6790/0 conn(0x7f686c13f710 legacy=0x7f686c0aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 >> v1:172.21.15.81:6789/0 conn(0x7f686c0a98f0 legacy=0x7f686c1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f6861ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f6861ffb640 1 monclient: found mon.a 2024-03-25T18:17:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f6861ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f6861ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:17:12.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686c0b16b0 con 0x7f686c1430c0 2024-03-25T18:17:12.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f6861ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:12.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f6861ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:12.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f6861ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:12.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f68734d9640 5 monclient: authenticate success, global_id 4413 2024-03-25T18:17:12.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f68734d9640 10 monclient: _renew_subs 2024-03-25T18:17:12.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f68734d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:17:12.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f68734d9640 1 -- 172.21.15.63:0/2912791461 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f686c0b0c10 con 0x7f686c1430c0 2024-03-25T18:17:12.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f68734d9640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:12.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f68734d9640 10 monclient: _renew_subs 2024-03-25T18:17:12.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f68734d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:17:12.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f68734d9640 1 -- 172.21.15.63:0/2912791461 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f686c0b0fb0 con 0x7f686c1430c0 2024-03-25T18:17:12.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.487+0000 7f68734d9640 1 librados: init done 2024-03-25T18:17:12.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.488+0000 7f68734d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:12.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.488+0000 7f68734d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:12.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.488+0000 7f68734d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:17:12.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.488+0000 7f68734d9640 1 -- 172.21.15.63:0/2912791461 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f686c1d4ce0 con 0x7f686c1430c0 2024-03-25T18:17:12.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.808+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68640020a0 con 0x7f686c1430c0 2024-03-25T18:17:12.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.808+0000 7f6861ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:12.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.808+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6864003dc0 con 0x7f686c1430c0 2024-03-25T18:17:12.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.808+0000 7f6861ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:12.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.808+0000 7f6861ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:12.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.808+0000 7f6861ffb640 10 monclient: dump: 2024-03-25T18:17:12.809 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:12.809 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:12.809 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:12.810 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:12.810 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:12.810 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:12.810 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:12.810 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:12.810 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:12.810 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:12.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.808+0000 7f68637fe640 4 set_mon_vals no callback set 2024-03-25T18:17:12.811 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.808+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f6864019d90 con 0x7f686c1430c0 2024-03-25T18:17:12.811 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.809+0000 7f6871275640 10 monclient: get_auth_request con 0x7f685006dda0 auth_method 0 2024-03-25T18:17:12.811 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.809+0000 7f6861ffb640 10 client.4413.objecter ms_handle_connect 0x7f685006dda0 2024-03-25T18:17:12.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.829+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f68640539d0 con 0x7f686c1430c0 2024-03-25T18:17:12.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.829+0000 7f6861ffb640 10 client.4413.objecter ms_dispatch 0x7f686c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:12.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.829+0000 7f6861ffb640 3 client.4413.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:12.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.829+0000 7f6861ffb640 3 client.4413.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:12.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.829+0000 7f6861ffb640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-25T18:17:12.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.829+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f686404f310 con 0x7f686c1430c0 2024-03-25T18:17:12.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.829+0000 7f6861ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:12.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:12.829+0000 7f6861ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:13.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.087+0000 7f68734d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:13.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.087+0000 7f68734d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:13.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.087+0000 7f68734d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:17:13.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.087+0000 7f68734d9640 1 -- 172.21.15.63:0/2912791461 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f686c0627b0 con 0x7f686c1430c0 2024-03-25T18:17:13.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.116+0000 7f6861ffb640 1 -- 172.21.15.63:0/2912791461 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f6864051410 con 0x7f686c1430c0 2024-03-25T18:17:13.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.116+0000 7f6861ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:13.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.116+0000 7f6861ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:13.118 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:17:13.118 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:13.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.118+0000 7f68734d9640 10 librados: watch_flush enter 2024-03-25T18:17:13.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.118+0000 7f68734d9640 10 librados: watch_flush exit 2024-03-25T18:17:13.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.118+0000 7f68734d9640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-25T18:17:13.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.118+0000 7f68734d9640 10 client.4413.objecter successfully canceled tick 2024-03-25T18:17:13.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.118+0000 7f68734d9640 1 -- 172.21.15.63:0/2912791461 >> v1:172.21.15.81:6800/3640340873 conn(0x7f685006dda0 legacy=0x7f6850070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.118+0000 7f68734d9640 10 monclient: shutdown 2024-03-25T18:17:13.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.118+0000 7f68734d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:13.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.118+0000 7f68734d9640 1 -- 172.21.15.63:0/2912791461 >> v1:172.21.15.63:6789/0 conn(0x7f686c1430c0 legacy=0x7f686c1d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.118+0000 7f68734d9640 1 -- 172.21.15.63:0/2912791461 shutdown_connections 2024-03-25T18:17:13.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.118+0000 7f68734d9640 1 -- 172.21.15.63:0/2912791461 >> 172.21.15.63:0/2912791461 conn(0x7f686c0b7ea0 msgr2=0x7f686c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:13.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.118+0000 7f68734d9640 1 -- 172.21.15.63:0/2912791461 shutdown_connections 2024-03-25T18:17:13.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.118+0000 7f68734d9640 1 -- 172.21.15.63:0/2912791461 wait complete. 2024-03-25T18:17:13.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:13.118+0000 7f68734d9640 1 librados: shutdown 2024-03-25T18:17:13.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:18.909 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:00.601 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:18.915 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:00.608 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:18.937 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:00.629 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:18.962 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:00.654 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:18.989 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:05:00.649 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:18.996 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:00.689 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:19.006 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:05:00.666 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:19.022 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:00.714 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:19.032 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:05:00.693 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:19.937 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:20.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.040+0000 7fc4cb693640 10 monclient: get_monmap_and_config 2024-03-25T18:17:20.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.040+0000 7fc4cb693640 10 monclient: build_initial_monmap 2024-03-25T18:17:20.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.040+0000 7fc4cb693640 10 monclient: monmap: 2024-03-25T18:17:20.042 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:17:20.042 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:20.042 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:17:20.041943+0000 2024-03-25T18:17:20.042 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:17:20.041943+0000 2024-03-25T18:17:20.042 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:20.042 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:20.042 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:17:20.042 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:17:20.042 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:17:20.042 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:20.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.041+0000 7fc4cb693640 1 Processor -- start 2024-03-25T18:17:20.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.041+0000 7fc4cb693640 1 -- start start 2024-03-25T18:17:20.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.041+0000 7fc4cb693640 10 monclient: init 2024-03-25T18:17:20.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.042+0000 7fc4cb693640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:20.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.042+0000 7fc4cb693640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:20.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.042+0000 7fc4cb693640 10 monclient(hunting): picked mon.noname-b con 0x7fc4c40b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T18:17:20.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.042+0000 7fc4cb693640 10 monclient(hunting): picked mon.noname-c con 0x7fc4c40b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:17:20.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.042+0000 7fc4cb693640 10 monclient(hunting): picked mon.noname-a con 0x7fc4c414be90 addr v1:172.21.15.63:6789/0 2024-03-25T18:17:20.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.042+0000 7fc4cb693640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c40af940 con 0x7fc4c414be90 2024-03-25T18:17:20.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.042+0000 7fc4cb693640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c40b0a70 con 0x7fc4c40b1730 2024-03-25T18:17:20.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.042+0000 7fc4cb693640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c40b0d50 con 0x7fc4c40b4d60 2024-03-25T18:17:20.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.042+0000 7fc4cb693640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:20.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.042+0000 7fc4cb693640 10 monclient(hunting): authenticate will time out at 7734.228485s 2024-03-25T18:17:20.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.042+0000 7fc4c942f640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fc4c40b4d60 0x7fc4c413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:40658/0 (socket says 172.21.15.63:40658) 2024-03-25T18:17:20.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.042+0000 7fc4c942f640 1 -- 172.21.15.63:0/1265903469 learned_addr learned my addr 172.21.15.63:0/1265903469 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:17:20.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.091+0000 7fc4b7fff640 1 -- 172.21.15.63:0/1265903469 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298948864 0 0) 0x7fc4c40b0a70 con 0x7fc4c40b1730 2024-03-25T18:17:20.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.091+0000 7fc4b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:20.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.091+0000 7fc4b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:20.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.091+0000 7fc4b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:20.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.091+0000 7fc4b7fff640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:17:20.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.091+0000 7fc4b7fff640 1 -- 172.21.15.63:0/1265903469 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4b0003520 con 0x7fc4c40b1730 2024-03-25T18:17:20.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.095+0000 7fc4b7fff640 1 -- 172.21.15.63:0/1265903469 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363376261 0 0) 0x7fc4c40af940 con 0x7fc4c414be90 2024-03-25T18:17:20.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.095+0000 7fc4b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:20.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.095+0000 7fc4b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:20.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.095+0000 7fc4b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:20.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.095+0000 7fc4b7fff640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:17:20.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.095+0000 7fc4b7fff640 1 -- 172.21.15.63:0/1265903469 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4c40b0a70 con 0x7fc4c414be90 2024-03-25T18:17:20.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4b7fff640 1 -- 172.21.15.63:0/1265903469 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1341692241 0 0) 0x7fc4b0003520 con 0x7fc4c40b1730 2024-03-25T18:17:20.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4b7fff640 1 -- 172.21.15.63:0/1265903469 >> v1:172.21.15.63:6790/0 conn(0x7fc4c40b4d60 legacy=0x7fc4c413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:20.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4b7fff640 1 -- 172.21.15.63:0/1265903469 >> v1:172.21.15.63:6789/0 conn(0x7fc4c414be90 legacy=0x7fc4c414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:20.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4b7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:20.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4b7fff640 1 monclient: found mon.noname-c 2024-03-25T18:17:20.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:20.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4b7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:17:20.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4b7fff640 1 -- 172.21.15.63:0/1265903469 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c40b1030 con 0x7fc4c40b1730 2024-03-25T18:17:20.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4b7fff640 10 monclient: _finish_auth 0 2024-03-25T18:17:20.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4b7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:20.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:20.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4b7fff640 1 -- 172.21.15.63:0/1265903469 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc4b8001690 con 0x7fc4c40b1730 2024-03-25T18:17:20.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4cb693640 5 monclient: authenticate success, global_id 4441 2024-03-25T18:17:20.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4cb693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:20.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:20.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4b7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:20.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4b7fff640 10 monclient: dump: 2024-03-25T18:17:20.102 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:20.102 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:20.102 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:20.102 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:20.102 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:20.103 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:20.103 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:20.103 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:20.103 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:20.103 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:20.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.100+0000 7fc4cb693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:20.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.114+0000 7fc4b7fff640 1 -- 172.21.15.63:0/1265903469 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4b80017f0 con 0x7fc4c40b1730 2024-03-25T18:17:20.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.114+0000 7fc4b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:20.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.114+0000 7fc4b7fff640 1 -- 172.21.15.63:0/1265903469 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc4b8001c00 con 0x7fc4c40b1730 2024-03-25T18:17:20.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.114+0000 7fc4cb693640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:20.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.114+0000 7fc4cb693640 4 set_mon_vals no callback set 2024-03-25T18:17:20.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.114+0000 7fc4b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:20.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.114+0000 7fc4b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:20.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.114+0000 7fc4b7fff640 10 monclient: dump: 2024-03-25T18:17:20.116 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:20.116 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:20.116 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:20.116 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:20.116 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:20.116 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:20.116 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:20.116 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:20.116 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:20.116 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:20.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.115+0000 7fc4cb693640 10 monclient: shutdown 2024-03-25T18:17:20.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.115+0000 7fc4cb693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:20.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.115+0000 7fc4cb693640 1 -- 172.21.15.63:0/1265903469 >> v1:172.21.15.81:6789/0 conn(0x7fc4c40b1730 legacy=0x7fc4c40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:20.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.115+0000 7fc4cb693640 1 -- 172.21.15.63:0/1265903469 shutdown_connections 2024-03-25T18:17:20.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.115+0000 7fc4cb693640 1 -- 172.21.15.63:0/1265903469 >> 172.21.15.63:0/1265903469 conn(0x7fc4c40617a0 msgr2=0x7fc4c4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:20.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.115+0000 7fc4cb693640 1 -- 172.21.15.63:0/1265903469 shutdown_connections 2024-03-25T18:17:20.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.115+0000 7fc4cb693640 1 -- 172.21.15.63:0/1265903469 wait complete. 2024-03-25T18:17:20.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.116+0000 7fc4cb693640 10 monclient: build_initial_monmap 2024-03-25T18:17:20.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.116+0000 7fc4cb693640 10 monclient: monmap: 2024-03-25T18:17:20.118 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:17:20.119 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:20.119 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:17:20.119 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:17:20.119 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:20.119 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:17:20.119 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:17:20.119 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:17:20.119 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:17:20.119 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:20.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.117+0000 7fc4cb693640 1 librados: starting msgr at 2024-03-25T18:17:20.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.117+0000 7fc4cb693640 1 librados: starting objecter 2024-03-25T18:17:20.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.117+0000 7fc4cb693640 1 Processor -- start 2024-03-25T18:17:20.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.117+0000 7fc4cb693640 1 -- start start 2024-03-25T18:17:20.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.117+0000 7fc4cb693640 1 librados: setting wanted keys 2024-03-25T18:17:20.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.117+0000 7fc4cb693640 1 librados: calling monclient init 2024-03-25T18:17:20.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.117+0000 7fc4cb693640 10 monclient: init 2024-03-25T18:17:20.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.118+0000 7fc4cb693640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:20.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.118+0000 7fc4cb693640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:20.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.118+0000 7fc4cb693640 10 monclient(hunting): picked mon.noname-b con 0x7fc4c40b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:17:20.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.118+0000 7fc4cb693640 10 monclient(hunting): picked mon.noname-a con 0x7fc4c40b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T18:17:20.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.118+0000 7fc4cb693640 10 monclient(hunting): picked mon.noname-c con 0x7fc4c414be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:17:20.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.118+0000 7fc4cb693640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c4137500 con 0x7fc4c40b4d60 2024-03-25T18:17:20.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.118+0000 7fc4cb693640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c41d7940 con 0x7fc4c40b1730 2024-03-25T18:17:20.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.118+0000 7fc4cb693640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c41d8570 con 0x7fc4c414be90 2024-03-25T18:17:20.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.118+0000 7fc4cb693640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:20.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.118+0000 7fc4cb693640 10 monclient(hunting): authenticate will time out at 7734.304673s 2024-03-25T18:17:20.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.140+0000 7fc4c942f640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc4c40b1730 0x7fc4c4135d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:59170/0 (socket says 172.21.15.63:59170) 2024-03-25T18:17:20.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.140+0000 7fc4c942f640 1 -- 172.21.15.63:0/2111149836 learned_addr learned my addr 172.21.15.63:0/2111149836 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:17:20.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.145+0000 7fc4b5ffb640 10 client.?.objecter ms_handle_connect 0x7fc4c40b1730 2024-03-25T18:17:20.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.145+0000 7fc4b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:20.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.155+0000 7fc4b5ffb640 10 client.?.objecter ms_handle_connect 0x7fc4c40b4d60 2024-03-25T18:17:20.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.155+0000 7fc4b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:20.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.159+0000 7fc4b5ffb640 10 client.?.objecter ms_handle_connect 0x7fc4c414be90 2024-03-25T18:17:20.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.159+0000 7fc4b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:20.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.160+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473500666 0 0) 0x7fc4c41d7940 con 0x7fc4c40b1730 2024-03-25T18:17:20.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.160+0000 7fc4b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:20.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.160+0000 7fc4b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:20.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.160+0000 7fc4b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:20.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.160+0000 7fc4b5ffb640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:17:20.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.160+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a8004eb0 con 0x7fc4c40b1730 2024-03-25T18:17:20.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.165+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826850926 0 0) 0x7fc4c41d8570 con 0x7fc4c414be90 2024-03-25T18:17:20.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.165+0000 7fc4b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:20.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.165+0000 7fc4b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:20.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.165+0000 7fc4b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:20.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.165+0000 7fc4b5ffb640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:17:20.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.165+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4c41d7940 con 0x7fc4c414be90 2024-03-25T18:17:20.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.170+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3608126362 0 0) 0x7fc4a8004eb0 con 0x7fc4c40b1730 2024-03-25T18:17:20.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.170+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4c41d8570 con 0x7fc4c40b1730 2024-03-25T18:17:20.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.170+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc4c00018b0 con 0x7fc4c40b1730 2024-03-25T18:17:20.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.170+0000 7fc4b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:20.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.170+0000 7fc4b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:20.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.170+0000 7fc4b5ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:20.172 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:20.172 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:20.172 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:20.172 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:20.172 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:20.172 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:20.172 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:20.172 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:20.173 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:20.173 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:20.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.170+0000 7fc4b5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:20.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.171+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734536602 0 0) 0x7fc4c4137500 con 0x7fc4c40b4d60 2024-03-25T18:17:20.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.171+0000 7fc4b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:20.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.171+0000 7fc4b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:20.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.171+0000 7fc4b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:20.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.171+0000 7fc4b5ffb640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:17:20.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.171+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a8004eb0 con 0x7fc4c40b4d60 2024-03-25T18:17:20.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.175+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 727141325 0 0) 0x7fc4c41d7940 con 0x7fc4c414be90 2024-03-25T18:17:20.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.175+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4c4137500 con 0x7fc4c414be90 2024-03-25T18:17:20.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.175+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc4bc001b90 con 0x7fc4c414be90 2024-03-25T18:17:20.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.175+0000 7fc4b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:20.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.175+0000 7fc4b5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:20.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.175+0000 7fc4b5ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:20.177 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:20.177 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:20.177 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:20.177 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:20.177 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:20.177 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:20.177 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:20.177 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:20.177 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:20.177 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:20.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.182+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 233581251 0 0) 0x7fc4c41d8570 con 0x7fc4c40b1730 2024-03-25T18:17:20.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.182+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 >> v1:172.21.15.63:6790/0 conn(0x7fc4c414be90 legacy=0x7fc4c41380a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:20.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.182+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 >> v1:172.21.15.63:6789/0 conn(0x7fc4c40b4d60 legacy=0x7fc4c41d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:20.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.182+0000 7fc4b5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:20.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.182+0000 7fc4b5ffb640 1 monclient: found mon.b 2024-03-25T18:17:20.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.183+0000 7fc4b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:20.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.183+0000 7fc4b5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:20.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.183+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c41d8700 con 0x7fc4c40b1730 2024-03-25T18:17:20.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.183+0000 7fc4b5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:20.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.183+0000 7fc4b5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:20.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.183+0000 7fc4b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:20.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.183+0000 7fc4cb693640 5 monclient: authenticate success, global_id 4444 2024-03-25T18:17:20.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.183+0000 7fc4cb693640 10 monclient: _renew_subs 2024-03-25T18:17:20.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.183+0000 7fc4cb693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:20.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.183+0000 7fc4cb693640 1 -- 172.21.15.63:0/2111149836 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c41387b0 con 0x7fc4c40b1730 2024-03-25T18:17:20.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.183+0000 7fc4cb693640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:20.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.183+0000 7fc4cb693640 10 monclient: _renew_subs 2024-03-25T18:17:20.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.183+0000 7fc4cb693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:20.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.183+0000 7fc4cb693640 1 -- 172.21.15.63:0/2111149836 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c4137b00 con 0x7fc4c40b1730 2024-03-25T18:17:20.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.183+0000 7fc4cb693640 1 librados: init done 2024-03-25T18:17:20.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.184+0000 7fc4cb693640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:20.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.184+0000 7fc4cb693640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:20.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.184+0000 7fc4cb693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:20.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.184+0000 7fc4cb693640 1 -- 172.21.15.63:0/2111149836 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c40627b0 con 0x7fc4c40b1730 2024-03-25T18:17:20.461 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.460+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4c00047e0 con 0x7fc4c40b1730 2024-03-25T18:17:20.461 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.460+0000 7fc4b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:20.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.460+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc4c0004c30 con 0x7fc4c40b1730 2024-03-25T18:17:20.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.460+0000 7fc4b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:20.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.460+0000 7fc4b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:20.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.460+0000 7fc4b5ffb640 10 monclient: dump: 2024-03-25T18:17:20.462 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:20.462 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:20.462 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:20.462 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:20.462 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:20.462 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:20.462 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:20.462 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:20.462 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:20.463 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:20.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.460+0000 7fc4b77fe640 4 set_mon_vals no callback set 2024-03-25T18:17:20.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.461+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fc4c0004e50 con 0x7fc4c40b1730 2024-03-25T18:17:20.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.462+0000 7fc4c8c2e640 10 monclient: get_auth_request con 0x7fc4a806dfb0 auth_method 0 2024-03-25T18:17:20.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.462+0000 7fc4b5ffb640 10 client.4444.objecter ms_handle_connect 0x7fc4a806dfb0 2024-03-25T18:17:20.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.467+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fc4c0054810 con 0x7fc4c40b1730 2024-03-25T18:17:20.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.467+0000 7fc4b5ffb640 10 client.4444.objecter ms_dispatch 0x7fc4c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:20.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.467+0000 7fc4b5ffb640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:20.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.467+0000 7fc4b5ffb640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:20.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.467+0000 7fc4b5ffb640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-25T18:17:20.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.468+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4c0050320 con 0x7fc4c40b1730 2024-03-25T18:17:20.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.468+0000 7fc4b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:20.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.468+0000 7fc4b5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:20.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.727+0000 7fc4cb693640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:20.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.727+0000 7fc4cb693640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:20.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.727+0000 7fc4cb693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:20.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.727+0000 7fc4cb693640 1 -- 172.21.15.63:0/2111149836 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4c4069f40 con 0x7fc4c40b1730 2024-03-25T18:17:20.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.748+0000 7fc4b5ffb640 1 -- 172.21.15.63:0/2111149836 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fc4c0052250 con 0x7fc4c40b1730 2024-03-25T18:17:20.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.748+0000 7fc4b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:20.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.748+0000 7fc4b5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:20.750 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:17:20.750 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:20.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.751+0000 7fc4cb693640 10 librados: watch_flush enter 2024-03-25T18:17:20.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.751+0000 7fc4cb693640 10 librados: watch_flush exit 2024-03-25T18:17:20.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.751+0000 7fc4cb693640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-25T18:17:20.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.751+0000 7fc4cb693640 10 client.4444.objecter successfully canceled tick 2024-03-25T18:17:20.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.751+0000 7fc4cb693640 1 -- 172.21.15.63:0/2111149836 >> v1:172.21.15.81:6800/3640340873 conn(0x7fc4a806dfb0 legacy=0x7fc4a8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:20.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.751+0000 7fc4cb693640 10 monclient: shutdown 2024-03-25T18:17:20.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.751+0000 7fc4cb693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:20.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.751+0000 7fc4cb693640 1 -- 172.21.15.63:0/2111149836 >> v1:172.21.15.81:6789/0 conn(0x7fc4c40b1730 legacy=0x7fc4c4135d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:20.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.751+0000 7fc4cb693640 1 -- 172.21.15.63:0/2111149836 shutdown_connections 2024-03-25T18:17:20.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.751+0000 7fc4cb693640 1 -- 172.21.15.63:0/2111149836 >> 172.21.15.63:0/2111149836 conn(0x7fc4c40617a0 msgr2=0x7fc4c4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:20.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.751+0000 7fc4cb693640 1 -- 172.21.15.63:0/2111149836 shutdown_connections 2024-03-25T18:17:20.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.751+0000 7fc4cb693640 1 -- 172.21.15.63:0/2111149836 wait complete. 2024-03-25T18:17:20.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:20.751+0000 7fc4cb693640 1 librados: shutdown 2024-03-25T18:17:20.764 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:27.572 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:27.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.684+0000 7f8bab9cd640 10 monclient: get_monmap_and_config 2024-03-25T18:17:27.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.684+0000 7f8bab9cd640 10 monclient: build_initial_monmap 2024-03-25T18:17:27.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.684+0000 7f8bab9cd640 10 monclient: monmap: 2024-03-25T18:17:27.686 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:17:27.686 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:27.686 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:17:27.685166+0000 2024-03-25T18:17:27.686 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:17:27.685166+0000 2024-03-25T18:17:27.687 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:27.687 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:27.687 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:17:27.687 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:17:27.687 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:17:27.687 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:27.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.684+0000 7f8bab9cd640 1 Processor -- start 2024-03-25T18:17:27.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.685+0000 7f8bab9cd640 1 -- start start 2024-03-25T18:17:27.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.685+0000 7f8bab9cd640 10 monclient: init 2024-03-25T18:17:27.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.685+0000 7f8bab9cd640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:27.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.685+0000 7f8bab9cd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:27.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.685+0000 7f8bab9cd640 10 monclient(hunting): picked mon.noname-b con 0x7f8ba40a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:17:27.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.685+0000 7f8bab9cd640 10 monclient(hunting): picked mon.noname-c con 0x7f8ba413f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:17:27.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.685+0000 7f8bab9cd640 10 monclient(hunting): picked mon.noname-a con 0x7f8ba41430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:17:27.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.685+0000 7f8bab9cd640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba4142520 con 0x7f8ba41430c0 2024-03-25T18:17:27.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.685+0000 7f8bab9cd640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba40b28a0 con 0x7f8ba413f710 2024-03-25T18:17:27.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.685+0000 7f8bab9cd640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba40b2a30 con 0x7f8ba40a98f0 2024-03-25T18:17:27.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.685+0000 7f8bab9cd640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:27.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.685+0000 7f8bab9cd640 10 monclient(hunting): authenticate will time out at 7741.872052s 2024-03-25T18:17:27.696 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.695+0000 7f8ba9f6a640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f8ba41430c0 0x7f8ba414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:51506/0 (socket says 172.21.15.63:51506) 2024-03-25T18:17:27.696 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.695+0000 7f8ba9f6a640 1 -- 172.21.15.63:0/199592442 learned_addr learned my addr 172.21.15.63:0/199592442 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:17:27.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.702+0000 7f8b9bfff640 1 -- 172.21.15.63:0/199592442 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577701067 0 0) 0x7f8ba4142520 con 0x7f8ba41430c0 2024-03-25T18:17:27.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.702+0000 7f8b9bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:27.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.702+0000 7f8b9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:27.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.702+0000 7f8b9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:27.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.702+0000 7f8b9bfff640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:17:27.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.702+0000 7f8b9bfff640 1 -- 172.21.15.63:0/199592442 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b90003520 con 0x7f8ba41430c0 2024-03-25T18:17:27.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8b9bfff640 1 -- 172.21.15.63:0/199592442 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3393982154 0 0) 0x7f8b90003520 con 0x7f8ba41430c0 2024-03-25T18:17:27.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8b9bfff640 1 -- 172.21.15.63:0/199592442 >> v1:172.21.15.63:6790/0 conn(0x7f8ba40a98f0 legacy=0x7f8ba40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:27.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8b9bfff640 1 -- 172.21.15.63:0/199592442 >> v1:172.21.15.81:6789/0 conn(0x7f8ba413f710 legacy=0x7f8ba41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:27.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8b9bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:27.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8b9bfff640 1 monclient: found mon.noname-a 2024-03-25T18:17:27.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8b9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:27.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8b9bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:17:27.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8b9bfff640 1 -- 172.21.15.63:0/199592442 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba40b44c0 con 0x7f8ba41430c0 2024-03-25T18:17:27.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8b9bfff640 10 monclient: _finish_auth 0 2024-03-25T18:17:27.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8b9bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:27.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8b9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:27.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8b9bfff640 1 -- 172.21.15.63:0/199592442 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8b9c001a10 con 0x7f8ba41430c0 2024-03-25T18:17:27.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8b9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:27.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8b9bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:27.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8b9bfff640 10 monclient: dump: 2024-03-25T18:17:27.711 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:27.711 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:27.711 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:27.711 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:27.711 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:27.711 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:27.711 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:27.711 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:27.711 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:27.711 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:27.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8bab9cd640 5 monclient: authenticate success, global_id 4422 2024-03-25T18:17:27.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.708+0000 7f8bab9cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:27.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.713+0000 7f8b9bfff640 1 -- 172.21.15.63:0/199592442 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b9c001b70 con 0x7f8ba41430c0 2024-03-25T18:17:27.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.713+0000 7f8b9bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:27.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.713+0000 7f8b9bfff640 1 -- 172.21.15.63:0/199592442 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8b9c001f80 con 0x7f8ba41430c0 2024-03-25T18:17:27.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.713+0000 7f8bab9cd640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:27.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.713+0000 7f8bab9cd640 4 set_mon_vals no callback set 2024-03-25T18:17:27.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.713+0000 7f8b9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:27.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.713+0000 7f8b9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:27.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.713+0000 7f8b9bfff640 10 monclient: dump: 2024-03-25T18:17:27.715 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:27.715 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:27.715 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:27.715 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:27.715 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:27.715 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:27.715 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:27.715 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:27.715 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:27.715 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:27.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.713+0000 7f8bab9cd640 10 monclient: shutdown 2024-03-25T18:17:27.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.713+0000 7f8bab9cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:27.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.713+0000 7f8bab9cd640 1 -- 172.21.15.63:0/199592442 >> v1:172.21.15.63:6789/0 conn(0x7f8ba41430c0 legacy=0x7f8ba414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:27.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.713+0000 7f8bab9cd640 1 -- 172.21.15.63:0/199592442 shutdown_connections 2024-03-25T18:17:27.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.713+0000 7f8bab9cd640 1 -- 172.21.15.63:0/199592442 >> 172.21.15.63:0/199592442 conn(0x7f8ba40b7ea0 msgr2=0x7f8ba40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:27.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.713+0000 7f8bab9cd640 1 -- 172.21.15.63:0/199592442 shutdown_connections 2024-03-25T18:17:27.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.713+0000 7f8bab9cd640 1 -- 172.21.15.63:0/199592442 wait complete. 2024-03-25T18:17:27.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.714+0000 7f8bab9cd640 10 monclient: build_initial_monmap 2024-03-25T18:17:27.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.714+0000 7f8bab9cd640 10 monclient: monmap: 2024-03-25T18:17:27.718 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:17:27.719 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:27.719 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:17:27.719 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:17:27.719 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:27.719 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:17:27.719 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:17:27.719 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:17:27.719 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:17:27.719 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:27.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 1 librados: starting msgr at 2024-03-25T18:17:27.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 1 librados: starting objecter 2024-03-25T18:17:27.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 1 Processor -- start 2024-03-25T18:17:27.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 1 -- start start 2024-03-25T18:17:27.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 1 librados: setting wanted keys 2024-03-25T18:17:27.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 1 librados: calling monclient init 2024-03-25T18:17:27.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 10 monclient: init 2024-03-25T18:17:27.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:27.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:27.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 10 monclient(hunting): picked mon.noname-b con 0x7f8ba40a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:17:27.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 10 monclient(hunting): picked mon.noname-a con 0x7f8ba413f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:17:27.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 10 monclient(hunting): picked mon.noname-c con 0x7f8ba41430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:17:27.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba406fbb0 con 0x7f8ba413f710 2024-03-25T18:17:27.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba406fe20 con 0x7f8ba40a98f0 2024-03-25T18:17:27.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba4070100 con 0x7f8ba41430c0 2024-03-25T18:17:27.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:27.721 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.715+0000 7f8bab9cd640 10 monclient(hunting): authenticate will time out at 7741.902151s 2024-03-25T18:17:27.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.730+0000 7f8ba8f68640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f8ba413f710 0x7f8ba40712f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:51550/0 (socket says 172.21.15.63:51550) 2024-03-25T18:17:27.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.730+0000 7f8ba8f68640 1 -- 172.21.15.63:0/1909220906 learned_addr learned my addr 172.21.15.63:0/1909220906 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:17:27.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.731+0000 7f8b99ffb640 10 client.?.objecter ms_handle_connect 0x7f8ba413f710 2024-03-25T18:17:27.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.731+0000 7f8b99ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:27.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.732+0000 7f8b99ffb640 10 client.?.objecter ms_handle_connect 0x7f8ba41430c0 2024-03-25T18:17:27.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.732+0000 7f8b99ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:27.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.737+0000 7f8b99ffb640 10 client.?.objecter ms_handle_connect 0x7f8ba40a98f0 2024-03-25T18:17:27.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.737+0000 7f8b99ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:27.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.748+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041693841 0 0) 0x7f8ba4070100 con 0x7f8ba41430c0 2024-03-25T18:17:27.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.748+0000 7f8b99ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:27.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.748+0000 7f8b99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:27.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.748+0000 7f8b99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:27.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.748+0000 7f8b99ffb640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:17:27.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.749+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b88004eb0 con 0x7f8ba41430c0 2024-03-25T18:17:27.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.750+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440327672 0 0) 0x7f8ba406fe20 con 0x7f8ba40a98f0 2024-03-25T18:17:27.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.750+0000 7f8b99ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:27.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.750+0000 7f8b99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:27.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.750+0000 7f8b99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:27.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.750+0000 7f8b99ffb640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:17:27.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.750+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ba4070100 con 0x7f8ba40a98f0 2024-03-25T18:17:27.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.750+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849138694 0 0) 0x7f8ba406fbb0 con 0x7f8ba413f710 2024-03-25T18:17:27.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.750+0000 7f8b99ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:27.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.750+0000 7f8b99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:27.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.750+0000 7f8b99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:27.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.750+0000 7f8b99ffb640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:17:27.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.750+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ba406fe20 con 0x7f8ba413f710 2024-03-25T18:17:27.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.761+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3527419256 0 0) 0x7f8b88004eb0 con 0x7f8ba41430c0 2024-03-25T18:17:27.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.761+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ba406fbb0 con 0x7f8ba41430c0 2024-03-25T18:17:27.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.761+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8b9c001e80 con 0x7f8ba41430c0 2024-03-25T18:17:27.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.761+0000 7f8b99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:27.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.761+0000 7f8b99ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:27.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.761+0000 7f8b99ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:27.762 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:27.763 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:27.763 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:27.763 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:27.763 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:27.763 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:27.763 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:27.763 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:27.763 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:27.763 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:27.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.761+0000 7f8b99ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:27.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.766+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1670490448 0 0) 0x7f8ba406fbb0 con 0x7f8ba41430c0 2024-03-25T18:17:27.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.766+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 >> v1:172.21.15.81:6789/0 conn(0x7f8ba40a98f0 legacy=0x7f8ba4070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:27.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.766+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 >> v1:172.21.15.63:6789/0 conn(0x7f8ba413f710 legacy=0x7f8ba40712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:27.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.766+0000 7f8b99ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:27.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.766+0000 7f8b99ffb640 1 monclient: found mon.c 2024-03-25T18:17:27.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.766+0000 7f8b99ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:27.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.766+0000 7f8b99ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:17:27.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.766+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba40703e0 con 0x7f8ba41430c0 2024-03-25T18:17:27.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.766+0000 7f8b99ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:27.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.766+0000 7f8b99ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:27.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.766+0000 7f8b99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:27.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.766+0000 7f8bab9cd640 5 monclient: authenticate success, global_id 4433 2024-03-25T18:17:27.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.766+0000 7f8bab9cd640 10 monclient: _renew_subs 2024-03-25T18:17:27.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.766+0000 7f8bab9cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:17:27.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.766+0000 7f8bab9cd640 1 -- 172.21.15.63:0/1909220906 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba406f240 con 0x7f8ba41430c0 2024-03-25T18:17:27.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.767+0000 7f8bab9cd640 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:27.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.767+0000 7f8bab9cd640 10 monclient: _renew_subs 2024-03-25T18:17:27.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.767+0000 7f8bab9cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:17:27.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.767+0000 7f8bab9cd640 1 -- 172.21.15.63:0/1909220906 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba406e2f0 con 0x7f8ba41430c0 2024-03-25T18:17:27.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.767+0000 7f8bab9cd640 1 librados: init done 2024-03-25T18:17:27.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.768+0000 7f8bab9cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:27.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.768+0000 7f8bab9cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:27.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.768+0000 7f8bab9cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:17:27.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:27.768+0000 7f8bab9cd640 1 -- 172.21.15.63:0/1909220906 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba40627b0 con 0x7f8ba41430c0 2024-03-25T18:17:28.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.074+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b9c002020 con 0x7f8ba41430c0 2024-03-25T18:17:28.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.075+0000 7f8b99ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:28.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.075+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8b9c004fb0 con 0x7f8ba41430c0 2024-03-25T18:17:28.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.075+0000 7f8b99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:28.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.075+0000 7f8b99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:28.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.075+0000 7f8b99ffb640 10 monclient: dump: 2024-03-25T18:17:28.077 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:28.077 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:28.077 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:28.077 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:28.077 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:28.077 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:28.077 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:28.078 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:28.078 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:28.078 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:28.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.075+0000 7f8b9b7fe640 4 set_mon_vals no callback set 2024-03-25T18:17:28.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.076+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f8b9c01af80 con 0x7f8ba41430c0 2024-03-25T18:17:28.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.077+0000 7f8ba9769640 10 monclient: get_auth_request con 0x7f8b8806dda0 auth_method 0 2024-03-25T18:17:28.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.077+0000 7f8b99ffb640 10 client.4433.objecter ms_handle_connect 0x7f8b8806dda0 2024-03-25T18:17:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.082+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f8b9c054b30 con 0x7f8ba41430c0 2024-03-25T18:17:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.082+0000 7f8b99ffb640 10 client.4433.objecter ms_dispatch 0x7f8ba4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.082+0000 7f8b99ffb640 3 client.4433.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.082+0000 7f8b99ffb640 3 client.4433.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.082+0000 7f8b99ffb640 20 client.4433.objecter dump_active .. 0 homeless 2024-03-25T18:17:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.082+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b9c050470 con 0x7f8ba41430c0 2024-03-25T18:17:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.082+0000 7f8b99ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:28.085 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.082+0000 7f8b99ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:28.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.339+0000 7f8bab9cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:28.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.339+0000 7f8bab9cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:28.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.339+0000 7f8bab9cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:17:28.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.339+0000 7f8bab9cd640 1 -- 172.21.15.63:0/1909220906 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ba406f3d0 con 0x7f8ba41430c0 2024-03-25T18:17:28.382 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.381+0000 7f8b99ffb640 1 -- 172.21.15.63:0/1909220906 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f8b9c052570 con 0x7f8ba41430c0 2024-03-25T18:17:28.382 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.381+0000 7f8b99ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:28.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.381+0000 7f8b99ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:28.383 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:17:28.383 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:28.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.383+0000 7f8bab9cd640 10 librados: watch_flush enter 2024-03-25T18:17:28.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.383+0000 7f8bab9cd640 10 librados: watch_flush exit 2024-03-25T18:17:28.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.383+0000 7f8bab9cd640 20 client.4433.objecter shutdown clearing up homeless session... 2024-03-25T18:17:28.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.383+0000 7f8bab9cd640 10 client.4433.objecter successfully canceled tick 2024-03-25T18:17:28.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.383+0000 7f8bab9cd640 1 -- 172.21.15.63:0/1909220906 >> v1:172.21.15.81:6800/3640340873 conn(0x7f8b8806dda0 legacy=0x7f8b88070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:28.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.383+0000 7f8bab9cd640 10 monclient: shutdown 2024-03-25T18:17:28.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.383+0000 7f8bab9cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:28.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.383+0000 7f8bab9cd640 1 -- 172.21.15.63:0/1909220906 >> v1:172.21.15.63:6790/0 conn(0x7f8ba41430c0 legacy=0x7f8ba41e6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:28.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.383+0000 7f8bab9cd640 1 -- 172.21.15.63:0/1909220906 shutdown_connections 2024-03-25T18:17:28.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.383+0000 7f8bab9cd640 1 -- 172.21.15.63:0/1909220906 >> 172.21.15.63:0/1909220906 conn(0x7f8ba40b7ea0 msgr2=0x7f8ba4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:28.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.383+0000 7f8bab9cd640 1 -- 172.21.15.63:0/1909220906 shutdown_connections 2024-03-25T18:17:28.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.384+0000 7f8bab9cd640 1 -- 172.21.15.63:0/1909220906 wait complete. 2024-03-25T18:17:28.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:28.384+0000 7f8bab9cd640 1 librados: shutdown 2024-03-25T18:17:28.396 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:35.203 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:35.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.309+0000 7f0ff3a2e640 10 monclient: get_monmap_and_config 2024-03-25T18:17:35.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.309+0000 7f0ff3a2e640 10 monclient: build_initial_monmap 2024-03-25T18:17:35.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.309+0000 7f0ff3a2e640 10 monclient: monmap: 2024-03-25T18:17:35.310 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:17:35.310 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:35.310 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:17:35.310188+0000 2024-03-25T18:17:35.310 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:17:35.310188+0000 2024-03-25T18:17:35.310 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:35.310 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:35.310 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:17:35.310 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:17:35.310 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:17:35.310 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:35.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.309+0000 7f0ff3a2e640 1 Processor -- start 2024-03-25T18:17:35.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.309+0000 7f0ff3a2e640 1 -- start start 2024-03-25T18:17:35.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.309+0000 7f0ff3a2e640 10 monclient: init 2024-03-25T18:17:35.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.310+0000 7f0ff3a2e640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:35.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.310+0000 7f0ff3a2e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:35.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.310+0000 7f0ff3a2e640 10 monclient(hunting): picked mon.noname-b con 0x7f0fec0b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T18:17:35.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.310+0000 7f0ff3a2e640 10 monclient(hunting): picked mon.noname-c con 0x7f0fec0b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:17:35.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.310+0000 7f0ff3a2e640 10 monclient(hunting): picked mon.noname-a con 0x7f0fec14be90 addr v1:172.21.15.63:6789/0 2024-03-25T18:17:35.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.310+0000 7f0ff3a2e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec0af940 con 0x7f0fec14be90 2024-03-25T18:17:35.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.310+0000 7f0ff3a2e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec0b0a70 con 0x7f0fec0b1730 2024-03-25T18:17:35.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.310+0000 7f0ff3a2e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec0b0d50 con 0x7f0fec0b4d60 2024-03-25T18:17:35.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.310+0000 7f0ff3a2e640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:35.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.310+0000 7f0ff3a2e640 10 monclient(hunting): authenticate will time out at 7749.496704s 2024-03-25T18:17:35.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.310+0000 7f0ff17ca640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f0fec0b4d60 0x7f0fec13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:47630/0 (socket says 172.21.15.63:47630) 2024-03-25T18:17:35.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.310+0000 7f0ff17ca640 1 -- 172.21.15.63:0/4230669569 learned_addr learned my addr 172.21.15.63:0/4230669569 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:17:35.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.375+0000 7f0fe3fff640 1 -- 172.21.15.63:0/4230669569 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017454567 0 0) 0x7f0fec0af940 con 0x7f0fec14be90 2024-03-25T18:17:35.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.375+0000 7f0fe3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:35.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.375+0000 7f0fe3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:35.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.375+0000 7f0fe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:35.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.375+0000 7f0fe3fff640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:17:35.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.376+0000 7f0fe3fff640 1 -- 172.21.15.63:0/4230669569 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd8003520 con 0x7f0fec14be90 2024-03-25T18:17:35.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0fe3fff640 1 -- 172.21.15.63:0/4230669569 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1820046735 0 0) 0x7f0fd8003520 con 0x7f0fec14be90 2024-03-25T18:17:35.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0fe3fff640 1 -- 172.21.15.63:0/4230669569 >> v1:172.21.15.63:6790/0 conn(0x7f0fec0b4d60 legacy=0x7f0fec13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:35.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0fe3fff640 1 -- 172.21.15.63:0/4230669569 >> v1:172.21.15.81:6789/0 conn(0x7f0fec0b1730 legacy=0x7f0fec0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:35.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0fe3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:35.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0fe3fff640 1 monclient: found mon.noname-a 2024-03-25T18:17:35.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0fe3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:35.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0fe3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:17:35.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0fe3fff640 1 -- 172.21.15.63:0/4230669569 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec0b1030 con 0x7f0fec14be90 2024-03-25T18:17:35.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0fe3fff640 10 monclient: _finish_auth 0 2024-03-25T18:17:35.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0fe3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:35.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0fe3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:35.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0fe3fff640 1 -- 172.21.15.63:0/4230669569 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f0fe4001a10 con 0x7f0fec14be90 2024-03-25T18:17:35.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0fe3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:35.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0fe3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:35.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0fe3fff640 10 monclient: dump: 2024-03-25T18:17:35.400 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:35.400 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:35.400 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:35.400 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:35.400 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:35.400 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:35.400 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:35.401 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:35.401 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:35.401 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:35.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0ff3a2e640 5 monclient: authenticate success, global_id 4428 2024-03-25T18:17:35.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.398+0000 7f0ff3a2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:35.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.430+0000 7f0fe3fff640 1 -- 172.21.15.63:0/4230669569 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fe4001b70 con 0x7f0fec14be90 2024-03-25T18:17:35.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.430+0000 7f0fe3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:35.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.430+0000 7f0fe3fff640 1 -- 172.21.15.63:0/4230669569 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f0fe4001f80 con 0x7f0fec14be90 2024-03-25T18:17:35.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.430+0000 7f0ff3a2e640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:35.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.430+0000 7f0ff3a2e640 4 set_mon_vals no callback set 2024-03-25T18:17:35.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.430+0000 7f0fe3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:35.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.430+0000 7f0fe3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:35.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.430+0000 7f0fe3fff640 10 monclient: dump: 2024-03-25T18:17:35.432 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:35.432 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:35.432 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:35.432 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:35.432 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:35.432 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:35.432 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:35.432 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:35.432 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:35.432 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:35.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.431+0000 7f0ff3a2e640 10 monclient: shutdown 2024-03-25T18:17:35.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.431+0000 7f0ff3a2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:35.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.431+0000 7f0ff3a2e640 1 -- 172.21.15.63:0/4230669569 >> v1:172.21.15.63:6789/0 conn(0x7f0fec14be90 legacy=0x7f0fec14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:35.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.431+0000 7f0ff3a2e640 1 -- 172.21.15.63:0/4230669569 shutdown_connections 2024-03-25T18:17:35.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.431+0000 7f0ff3a2e640 1 -- 172.21.15.63:0/4230669569 >> 172.21.15.63:0/4230669569 conn(0x7f0fec0617a0 msgr2=0x7f0fec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:35.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.431+0000 7f0ff3a2e640 1 -- 172.21.15.63:0/4230669569 shutdown_connections 2024-03-25T18:17:35.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.431+0000 7f0ff3a2e640 1 -- 172.21.15.63:0/4230669569 wait complete. 2024-03-25T18:17:35.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.432+0000 7f0ff3a2e640 10 monclient: build_initial_monmap 2024-03-25T18:17:35.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.432+0000 7f0ff3a2e640 10 monclient: monmap: 2024-03-25T18:17:35.435 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:17:35.435 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:35.435 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:17:35.436 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:17:35.436 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:35.436 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:17:35.436 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:17:35.436 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:17:35.436 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:17:35.436 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:35.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.432+0000 7f0ff3a2e640 1 librados: starting msgr at 2024-03-25T18:17:35.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.432+0000 7f0ff3a2e640 1 librados: starting objecter 2024-03-25T18:17:35.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.432+0000 7f0ff3a2e640 1 Processor -- start 2024-03-25T18:17:35.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.432+0000 7f0ff3a2e640 1 -- start start 2024-03-25T18:17:35.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.432+0000 7f0ff3a2e640 1 librados: setting wanted keys 2024-03-25T18:17:35.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.432+0000 7f0ff3a2e640 1 librados: calling monclient init 2024-03-25T18:17:35.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.432+0000 7f0ff3a2e640 10 monclient: init 2024-03-25T18:17:35.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.433+0000 7f0ff3a2e640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:35.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.433+0000 7f0ff3a2e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:35.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.433+0000 7f0ff3a2e640 10 monclient(hunting): picked mon.noname-a con 0x7f0fec0b1730 addr v1:172.21.15.63:6789/0 2024-03-25T18:17:35.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.433+0000 7f0ff3a2e640 10 monclient(hunting): picked mon.noname-c con 0x7f0fec0b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T18:17:35.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.433+0000 7f0ff3a2e640 10 monclient(hunting): picked mon.noname-b con 0x7f0fec14be90 addr v1:172.21.15.81:6789/0 2024-03-25T18:17:35.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.433+0000 7f0ff3a2e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec140810 con 0x7f0fec0b1730 2024-03-25T18:17:35.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.433+0000 7f0ff3a2e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec140380 con 0x7f0fec14be90 2024-03-25T18:17:35.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.433+0000 7f0ff3a2e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec140510 con 0x7f0fec0b4d60 2024-03-25T18:17:35.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.433+0000 7f0ff3a2e640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:35.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.433+0000 7f0ff3a2e640 10 monclient(hunting): authenticate will time out at 7749.619914s 2024-03-25T18:17:35.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.439+0000 7f0ff1fcb640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f0fec14be90 0x7f0fec1e31c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:34992/0 (socket says 172.21.15.63:34992) 2024-03-25T18:17:35.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.439+0000 7f0ff1fcb640 1 -- 172.21.15.63:0/3260205292 learned_addr learned my addr 172.21.15.63:0/3260205292 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:17:35.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.440+0000 7f0fe1ffb640 10 client.?.objecter ms_handle_connect 0x7f0fec14be90 2024-03-25T18:17:35.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.440+0000 7f0fe1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:35.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.446+0000 7f0fe1ffb640 10 client.?.objecter ms_handle_connect 0x7f0fec0b1730 2024-03-25T18:17:35.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.446+0000 7f0fe1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:35.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.447+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664090714 0 0) 0x7f0fec140380 con 0x7f0fec14be90 2024-03-25T18:17:35.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.447+0000 7f0fe1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:35.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.447+0000 7f0fe1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:35.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.447+0000 7f0fe1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:35.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.447+0000 7f0fe1ffb640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:17:35.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.447+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd0004eb0 con 0x7f0fec14be90 2024-03-25T18:17:35.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.461+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3076753324 0 0) 0x7f0fd0004eb0 con 0x7f0fec14be90 2024-03-25T18:17:35.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.461+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fec140380 con 0x7f0fec14be90 2024-03-25T18:17:35.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.461+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f0fe4001fc0 con 0x7f0fec14be90 2024-03-25T18:17:35.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.461+0000 7f0fe1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:35.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.461+0000 7f0fe1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:35.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.461+0000 7f0fe1ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:35.463 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:35.463 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:35.463 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:35.463 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:35.463 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:35.463 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:35.463 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:35.463 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:35.463 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:35.463 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:35.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.461+0000 7f0fe1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:35.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.462+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430002534 0 0) 0x7f0fec140810 con 0x7f0fec0b1730 2024-03-25T18:17:35.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.462+0000 7f0fe1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:35.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.462+0000 7f0fe1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:35.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.462+0000 7f0fe1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:35.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.462+0000 7f0fe1ffb640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:17:35.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.462+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd0004eb0 con 0x7f0fec0b1730 2024-03-25T18:17:35.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.463+0000 7f0fe1ffb640 10 client.?.objecter ms_handle_connect 0x7f0fec0b4d60 2024-03-25T18:17:35.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.463+0000 7f0fe1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:35.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.470+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1029366892 0 0) 0x7f0fec140380 con 0x7f0fec14be90 2024-03-25T18:17:35.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.470+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 >> v1:172.21.15.63:6790/0 conn(0x7f0fec0b4d60 legacy=0x7f0fec1d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:35.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.470+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 >> v1:172.21.15.63:6789/0 conn(0x7f0fec0b1730 legacy=0x7f0fec134ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:35.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.470+0000 7f0fe1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:35.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.470+0000 7f0fe1ffb640 1 monclient: found mon.b 2024-03-25T18:17:35.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.470+0000 7f0fe1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:35.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.470+0000 7f0fe1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:35.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.470+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec1355e0 con 0x7f0fec14be90 2024-03-25T18:17:35.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.470+0000 7f0fe1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:35.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.470+0000 7f0fe1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:35.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.470+0000 7f0fe1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:35.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.470+0000 7f0ff3a2e640 5 monclient: authenticate success, global_id 4456 2024-03-25T18:17:35.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.470+0000 7f0ff3a2e640 10 monclient: _renew_subs 2024-03-25T18:17:35.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.470+0000 7f0ff3a2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:35.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.470+0000 7f0ff3a2e640 1 -- 172.21.15.63:0/3260205292 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fec1357d0 con 0x7f0fec14be90 2024-03-25T18:17:35.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.471+0000 7f0ff3a2e640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:35.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.471+0000 7f0ff3a2e640 10 monclient: _renew_subs 2024-03-25T18:17:35.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.471+0000 7f0ff3a2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:35.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.471+0000 7f0ff3a2e640 1 -- 172.21.15.63:0/3260205292 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fec144da0 con 0x7f0fec14be90 2024-03-25T18:17:35.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.471+0000 7f0ff3a2e640 1 librados: init done 2024-03-25T18:17:35.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.472+0000 7f0ff3a2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:35.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.472+0000 7f0ff3a2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:35.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.472+0000 7f0ff3a2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:35.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.472+0000 7f0ff3a2e640 1 -- 172.21.15.63:0/3260205292 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fec141c10 con 0x7f0fec14be90 2024-03-25T18:17:35.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.755+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fe4003c20 con 0x7f0fec14be90 2024-03-25T18:17:35.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.755+0000 7f0fe1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:35.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.755+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f0fe4003ff0 con 0x7f0fec14be90 2024-03-25T18:17:35.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.755+0000 7f0fe1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:35.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.755+0000 7f0fe1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:35.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.755+0000 7f0fe1ffb640 10 monclient: dump: 2024-03-25T18:17:35.756 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:35.756 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:35.756 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:35.757 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:35.757 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:35.757 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:35.757 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:35.757 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:35.757 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:35.757 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:35.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.755+0000 7f0fe37fe640 4 set_mon_vals no callback set 2024-03-25T18:17:35.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.756+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f0fe40041d0 con 0x7f0fec14be90 2024-03-25T18:17:35.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.756+0000 7f0ff17ca640 10 monclient: get_auth_request con 0x7f0fd006de80 auth_method 0 2024-03-25T18:17:35.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.757+0000 7f0fe1ffb640 10 client.4456.objecter ms_handle_connect 0x7f0fd006de80 2024-03-25T18:17:35.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.761+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f0fe4054c70 con 0x7f0fec14be90 2024-03-25T18:17:35.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.761+0000 7f0fe1ffb640 10 client.4456.objecter ms_dispatch 0x7f0fec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:35.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.761+0000 7f0fe1ffb640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:35.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.761+0000 7f0fe1ffb640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:35.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.761+0000 7f0fe1ffb640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-25T18:17:35.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.761+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0fe4050780 con 0x7f0fec14be90 2024-03-25T18:17:35.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.761+0000 7f0fe1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:35.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:35.761+0000 7f0fe1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:36.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.001+0000 7f0ff3a2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:36.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.001+0000 7f0ff3a2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:36.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.001+0000 7f0ff3a2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:36.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.001+0000 7f0ff3a2e640 1 -- 172.21.15.63:0/3260205292 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fec0627b0 con 0x7f0fec14be90 2024-03-25T18:17:36.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.029+0000 7f0fe1ffb640 1 -- 172.21.15.63:0/3260205292 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f0fe40526b0 con 0x7f0fec14be90 2024-03-25T18:17:36.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.029+0000 7f0fe1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:36.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.029+0000 7f0fe1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:36.030 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:17:36.030 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:36.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.032+0000 7f0ff3a2e640 10 librados: watch_flush enter 2024-03-25T18:17:36.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.032+0000 7f0ff3a2e640 10 librados: watch_flush exit 2024-03-25T18:17:36.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.032+0000 7f0ff3a2e640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-25T18:17:36.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.032+0000 7f0ff3a2e640 10 client.4456.objecter successfully canceled tick 2024-03-25T18:17:36.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.032+0000 7f0ff3a2e640 1 -- 172.21.15.63:0/3260205292 >> v1:172.21.15.81:6800/3640340873 conn(0x7f0fd006de80 legacy=0x7f0fd0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.032+0000 7f0ff3a2e640 10 monclient: shutdown 2024-03-25T18:17:36.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.032+0000 7f0ff3a2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:36.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.032+0000 7f0ff3a2e640 1 -- 172.21.15.63:0/3260205292 >> v1:172.21.15.81:6789/0 conn(0x7f0fec14be90 legacy=0x7f0fec1e31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.033+0000 7f0ff3a2e640 1 -- 172.21.15.63:0/3260205292 shutdown_connections 2024-03-25T18:17:36.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.033+0000 7f0ff3a2e640 1 -- 172.21.15.63:0/3260205292 >> 172.21.15.63:0/3260205292 conn(0x7f0fec0617a0 msgr2=0x7f0fec0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:36.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.033+0000 7f0ff3a2e640 1 -- 172.21.15.63:0/3260205292 shutdown_connections 2024-03-25T18:17:36.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.033+0000 7f0ff3a2e640 1 -- 172.21.15.63:0/3260205292 wait complete. 2024-03-25T18:17:36.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:36.033+0000 7f0ff3a2e640 1 librados: shutdown 2024-03-25T18:17:36.045 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:38.971 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:20.663 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:38.978 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:20.670 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:39.030 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:20.722 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:39.032 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:05:20.693 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:39.043 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:05:20.704 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:39.052 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:20.745 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:39.055 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:20.748 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:39.060 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:05:20.721 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:39.080 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:20.772 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:42.852 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:42.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.968+0000 7f6c854b3640 10 monclient: get_monmap_and_config 2024-03-25T18:17:42.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.968+0000 7f6c854b3640 10 monclient: build_initial_monmap 2024-03-25T18:17:42.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.968+0000 7f6c854b3640 10 monclient: monmap: 2024-03-25T18:17:42.969 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:17:42.969 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:42.969 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:17:42.969368+0000 2024-03-25T18:17:42.970 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:17:42.969368+0000 2024-03-25T18:17:42.970 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:42.970 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:42.970 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:17:42.970 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:17:42.970 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:17:42.970 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:42.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.968+0000 7f6c854b3640 1 Processor -- start 2024-03-25T18:17:42.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.969+0000 7f6c854b3640 1 -- start start 2024-03-25T18:17:42.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.969+0000 7f6c854b3640 10 monclient: init 2024-03-25T18:17:42.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.969+0000 7f6c854b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:42.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.969+0000 7f6c854b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:42.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.969+0000 7f6c854b3640 10 monclient(hunting): picked mon.noname-c con 0x7f6c800a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:17:42.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.969+0000 7f6c854b3640 10 monclient(hunting): picked mon.noname-a con 0x7f6c8013f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:17:42.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.969+0000 7f6c854b3640 10 monclient(hunting): picked mon.noname-b con 0x7f6c801430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:17:42.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.969+0000 7f6c854b3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c80142520 con 0x7f6c8013f710 2024-03-25T18:17:42.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.969+0000 7f6c854b3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c800b28a0 con 0x7f6c800a98f0 2024-03-25T18:17:42.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.969+0000 7f6c854b3640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c800b2a30 con 0x7f6c801430c0 2024-03-25T18:17:42.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.969+0000 7f6c854b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:42.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.969+0000 7f6c854b3640 10 monclient(hunting): authenticate will time out at 7757.156106s 2024-03-25T18:17:42.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.983+0000 7f6c7f7fe640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f6c801430c0 0x7f6c8014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:47760/0 (socket says 172.21.15.63:47760) 2024-03-25T18:17:42.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:42.983+0000 7f6c7f7fe640 1 -- 172.21.15.63:0/745010762 learned_addr learned my addr 172.21.15.63:0/745010762 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:17:43.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.003+0000 7f6c7dffb640 1 -- 172.21.15.63:0/745010762 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 365214350 0 0) 0x7f6c800b2a30 con 0x7f6c801430c0 2024-03-25T18:17:43.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.003+0000 7f6c7dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:43.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.003+0000 7f6c7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:43.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.003+0000 7f6c7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:43.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.003+0000 7f6c7dffb640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:17:43.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.003+0000 7f6c7dffb640 1 -- 172.21.15.63:0/745010762 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c68003520 con 0x7f6c801430c0 2024-03-25T18:17:43.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.007+0000 7f6c7dffb640 1 -- 172.21.15.63:0/745010762 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637885086 0 0) 0x7f6c800b28a0 con 0x7f6c800a98f0 2024-03-25T18:17:43.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.007+0000 7f6c7dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:43.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.007+0000 7f6c7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:43.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.007+0000 7f6c7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:43.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.007+0000 7f6c7dffb640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:17:43.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.007+0000 7f6c7dffb640 1 -- 172.21.15.63:0/745010762 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c800b2a30 con 0x7f6c800a98f0 2024-03-25T18:17:43.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.013+0000 7f6c7dffb640 1 -- 172.21.15.63:0/745010762 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532713990 0 0) 0x7f6c80142520 con 0x7f6c8013f710 2024-03-25T18:17:43.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.014+0000 7f6c7dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:43.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.014+0000 7f6c7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:43.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.014+0000 7f6c7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:43.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.014+0000 7f6c7dffb640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:17:43.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.014+0000 7f6c7dffb640 1 -- 172.21.15.63:0/745010762 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c800b28a0 con 0x7f6c8013f710 2024-03-25T18:17:43.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c7dffb640 1 -- 172.21.15.63:0/745010762 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1345017385 0 0) 0x7f6c68003520 con 0x7f6c801430c0 2024-03-25T18:17:43.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c7dffb640 1 -- 172.21.15.63:0/745010762 >> v1:172.21.15.81:6789/0 conn(0x7f6c800a98f0 legacy=0x7f6c800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:43.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c7dffb640 1 -- 172.21.15.63:0/745010762 >> v1:172.21.15.63:6789/0 conn(0x7f6c8013f710 legacy=0x7f6c801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:43.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c7dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:43.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c7dffb640 1 monclient: found mon.noname-b 2024-03-25T18:17:43.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:43.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c7dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:17:43.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c7dffb640 1 -- 172.21.15.63:0/745010762 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c800b44c0 con 0x7f6c801430c0 2024-03-25T18:17:43.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c7dffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:43.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c7dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:43.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:43.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c7dffb640 1 -- 172.21.15.63:0/745010762 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6c78001a10 con 0x7f6c801430c0 2024-03-25T18:17:43.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c854b3640 5 monclient: authenticate success, global_id 4442 2024-03-25T18:17:43.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c854b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:43.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:43.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c7dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:43.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c7dffb640 10 monclient: dump: 2024-03-25T18:17:43.039 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:43.039 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:43.039 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:43.039 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:43.039 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:43.039 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:43.039 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:43.039 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:43.039 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:43.040 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:43.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.036+0000 7f6c854b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:43.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.064+0000 7f6c7dffb640 1 -- 172.21.15.63:0/745010762 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c78001b70 con 0x7f6c801430c0 2024-03-25T18:17:43.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.064+0000 7f6c7dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:43.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.064+0000 7f6c7dffb640 1 -- 172.21.15.63:0/745010762 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6c78001f80 con 0x7f6c801430c0 2024-03-25T18:17:43.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.064+0000 7f6c854b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:43.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.064+0000 7f6c854b3640 4 set_mon_vals no callback set 2024-03-25T18:17:43.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.064+0000 7f6c7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:43.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.064+0000 7f6c7dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:43.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.064+0000 7f6c7dffb640 10 monclient: dump: 2024-03-25T18:17:43.066 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:43.066 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:43.066 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:43.066 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:43.066 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:43.066 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:43.066 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:43.066 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:43.067 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:43.067 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:43.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.065+0000 7f6c854b3640 10 monclient: shutdown 2024-03-25T18:17:43.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.065+0000 7f6c854b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:43.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.065+0000 7f6c854b3640 1 -- 172.21.15.63:0/745010762 >> v1:172.21.15.63:6790/0 conn(0x7f6c801430c0 legacy=0x7f6c8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:43.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.065+0000 7f6c854b3640 1 -- 172.21.15.63:0/745010762 shutdown_connections 2024-03-25T18:17:43.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.065+0000 7f6c854b3640 1 -- 172.21.15.63:0/745010762 >> 172.21.15.63:0/745010762 conn(0x7f6c800b7ea0 msgr2=0x7f6c800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:43.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.065+0000 7f6c854b3640 1 -- 172.21.15.63:0/745010762 shutdown_connections 2024-03-25T18:17:43.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.065+0000 7f6c854b3640 1 -- 172.21.15.63:0/745010762 wait complete. 2024-03-25T18:17:43.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.066+0000 7f6c854b3640 10 monclient: build_initial_monmap 2024-03-25T18:17:43.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.066+0000 7f6c854b3640 10 monclient: monmap: 2024-03-25T18:17:43.068 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:17:43.069 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:43.069 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:17:43.069 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:17:43.069 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:43.069 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:17:43.069 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:17:43.069 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:17:43.069 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:17:43.069 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:43.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.066+0000 7f6c854b3640 1 librados: starting msgr at 2024-03-25T18:17:43.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.066+0000 7f6c854b3640 1 librados: starting objecter 2024-03-25T18:17:43.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.067+0000 7f6c854b3640 1 Processor -- start 2024-03-25T18:17:43.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.067+0000 7f6c854b3640 1 -- start start 2024-03-25T18:17:43.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.067+0000 7f6c854b3640 1 librados: setting wanted keys 2024-03-25T18:17:43.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.067+0000 7f6c854b3640 1 librados: calling monclient init 2024-03-25T18:17:43.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.067+0000 7f6c854b3640 10 monclient: init 2024-03-25T18:17:43.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.067+0000 7f6c854b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:43.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.067+0000 7f6c854b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:43.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.067+0000 7f6c854b3640 10 monclient(hunting): picked mon.noname-b con 0x7f6c800a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:17:43.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.067+0000 7f6c854b3640 10 monclient(hunting): picked mon.noname-c con 0x7f6c8013f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:17:43.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.067+0000 7f6c854b3640 10 monclient(hunting): picked mon.noname-a con 0x7f6c801430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:17:43.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.067+0000 7f6c854b3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c80138b40 con 0x7f6c801430c0 2024-03-25T18:17:43.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.067+0000 7f6c854b3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c80138e20 con 0x7f6c800a98f0 2024-03-25T18:17:43.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.067+0000 7f6c854b3640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c80139100 con 0x7f6c8013f710 2024-03-25T18:17:43.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.067+0000 7f6c854b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:43.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.067+0000 7f6c854b3640 10 monclient(hunting): authenticate will time out at 7757.254273s 2024-03-25T18:17:43.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.082+0000 7f6c7effd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f6c800a98f0 0x7f6c80137380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:53568/0 (socket says 172.21.15.63:53568) 2024-03-25T18:17:43.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.082+0000 7f6c7effd640 1 -- 172.21.15.63:0/116680904 learned_addr learned my addr 172.21.15.63:0/116680904 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:17:43.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.083+0000 7f6c677fe640 10 client.?.objecter ms_handle_connect 0x7f6c800a98f0 2024-03-25T18:17:43.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.083+0000 7f6c677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:43.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.087+0000 7f6c677fe640 1 -- 172.21.15.63:0/116680904 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703615042 0 0) 0x7f6c80138e20 con 0x7f6c800a98f0 2024-03-25T18:17:43.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.088+0000 7f6c677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:43.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.088+0000 7f6c677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:43.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.088+0000 7f6c677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:43.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.088+0000 7f6c677fe640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:17:43.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.088+0000 7f6c677fe640 1 -- 172.21.15.63:0/116680904 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c6c004eb0 con 0x7f6c800a98f0 2024-03-25T18:17:43.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.100+0000 7f6c677fe640 10 client.?.objecter ms_handle_connect 0x7f6c8013f710 2024-03-25T18:17:43.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.100+0000 7f6c677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:43.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.102+0000 7f6c677fe640 1 -- 172.21.15.63:0/116680904 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 764435952 0 0) 0x7f6c6c004eb0 con 0x7f6c800a98f0 2024-03-25T18:17:43.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.102+0000 7f6c677fe640 1 -- 172.21.15.63:0/116680904 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c80138e20 con 0x7f6c800a98f0 2024-03-25T18:17:43.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.102+0000 7f6c677fe640 1 -- 172.21.15.63:0/116680904 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6c70001a30 con 0x7f6c800a98f0 2024-03-25T18:17:43.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.102+0000 7f6c677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:43.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.102+0000 7f6c677fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:43.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.102+0000 7f6c677fe640 10 monclient(hunting): dump: 2024-03-25T18:17:43.103 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:43.103 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:43.103 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:43.103 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:43.103 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:43.103 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:43.103 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:43.104 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:43.104 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:43.104 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:43.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.102+0000 7f6c677fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:43.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.104+0000 7f6c677fe640 10 client.?.objecter ms_handle_connect 0x7f6c801430c0 2024-03-25T18:17:43.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.104+0000 7f6c677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:43.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.115+0000 7f6c677fe640 1 -- 172.21.15.63:0/116680904 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3291868396 0 0) 0x7f6c80138e20 con 0x7f6c800a98f0 2024-03-25T18:17:43.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c677fe640 1 -- 172.21.15.63:0/116680904 >> v1:172.21.15.63:6790/0 conn(0x7f6c8013f710 legacy=0x7f6c80137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:43.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c677fe640 1 -- 172.21.15.63:0/116680904 >> v1:172.21.15.63:6789/0 conn(0x7f6c801430c0 legacy=0x7f6c801e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:43.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c677fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:43.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c677fe640 1 monclient: found mon.b 2024-03-25T18:17:43.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:43.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c677fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:43.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c677fe640 1 -- 172.21.15.63:0/116680904 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c801397a0 con 0x7f6c800a98f0 2024-03-25T18:17:43.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c677fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:43.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c677fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:43.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:43.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c854b3640 5 monclient: authenticate success, global_id 4462 2024-03-25T18:17:43.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c854b3640 10 monclient: _renew_subs 2024-03-25T18:17:43.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c854b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:43.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c854b3640 1 -- 172.21.15.63:0/116680904 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c80136aa0 con 0x7f6c800a98f0 2024-03-25T18:17:43.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c854b3640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:43.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c854b3640 10 monclient: _renew_subs 2024-03-25T18:17:43.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c854b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:43.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c854b3640 1 -- 172.21.15.63:0/116680904 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8013c320 con 0x7f6c800a98f0 2024-03-25T18:17:43.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.116+0000 7f6c854b3640 1 librados: init done 2024-03-25T18:17:43.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.117+0000 7f6c854b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:43.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.117+0000 7f6c854b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:43.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.117+0000 7f6c854b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:43.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.117+0000 7f6c854b3640 1 -- 172.21.15.63:0/116680904 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c801395b0 con 0x7f6c800a98f0 2024-03-25T18:17:43.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.398+0000 7f6c677fe640 1 -- 172.21.15.63:0/116680904 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c70001d10 con 0x7f6c800a98f0 2024-03-25T18:17:43.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.398+0000 7f6c677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:43.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.398+0000 7f6c677fe640 1 -- 172.21.15.63:0/116680904 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6c70001b90 con 0x7f6c800a98f0 2024-03-25T18:17:43.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.398+0000 7f6c677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:43.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.398+0000 7f6c677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:43.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.398+0000 7f6c677fe640 10 monclient: dump: 2024-03-25T18:17:43.399 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:43.399 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:43.399 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:43.400 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:43.400 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:43.400 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:43.400 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:43.400 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:43.400 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:43.400 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:43.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.398+0000 7f6c7d7fa640 4 set_mon_vals no callback set 2024-03-25T18:17:43.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.399+0000 7f6c677fe640 1 -- 172.21.15.63:0/116680904 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f6c70003b20 con 0x7f6c800a98f0 2024-03-25T18:17:43.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.399+0000 7f6c7e7fc640 10 monclient: get_auth_request con 0x7f6c6c06dd00 auth_method 0 2024-03-25T18:17:43.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.400+0000 7f6c677fe640 10 client.4462.objecter ms_handle_connect 0x7f6c6c06dd00 2024-03-25T18:17:43.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.404+0000 7f6c677fe640 1 -- 172.21.15.63:0/116680904 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f6c700548d0 con 0x7f6c800a98f0 2024-03-25T18:17:43.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.404+0000 7f6c677fe640 10 client.4462.objecter ms_dispatch 0x7f6c80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:43.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.404+0000 7f6c677fe640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:43.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.404+0000 7f6c677fe640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:43.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.404+0000 7f6c677fe640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-25T18:17:43.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.405+0000 7f6c677fe640 1 -- 172.21.15.63:0/116680904 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6c700503e0 con 0x7f6c800a98f0 2024-03-25T18:17:43.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.405+0000 7f6c677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:43.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.405+0000 7f6c677fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:43.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.631+0000 7f6c854b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:43.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.631+0000 7f6c854b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:43.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.631+0000 7f6c854b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:43.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.631+0000 7f6c854b3640 1 -- 172.21.15.63:0/116680904 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c80139930 con 0x7f6c800a98f0 2024-03-25T18:17:43.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.662+0000 7f6c677fe640 1 -- 172.21.15.63:0/116680904 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f6c70052310 con 0x7f6c800a98f0 2024-03-25T18:17:43.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.662+0000 7f6c677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:43.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.662+0000 7f6c677fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:43.663 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:17:43.663 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:43.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.664+0000 7f6c854b3640 10 librados: watch_flush enter 2024-03-25T18:17:43.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.664+0000 7f6c854b3640 10 librados: watch_flush exit 2024-03-25T18:17:43.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.664+0000 7f6c854b3640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-25T18:17:43.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.664+0000 7f6c854b3640 10 client.4462.objecter successfully canceled tick 2024-03-25T18:17:43.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.664+0000 7f6c854b3640 1 -- 172.21.15.63:0/116680904 >> v1:172.21.15.81:6800/3640340873 conn(0x7f6c6c06dd00 legacy=0x7f6c6c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:43.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.664+0000 7f6c854b3640 10 monclient: shutdown 2024-03-25T18:17:43.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.664+0000 7f6c854b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:43.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.664+0000 7f6c854b3640 1 -- 172.21.15.63:0/116680904 >> v1:172.21.15.81:6789/0 conn(0x7f6c800a98f0 legacy=0x7f6c80137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:43.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.664+0000 7f6c854b3640 1 -- 172.21.15.63:0/116680904 shutdown_connections 2024-03-25T18:17:43.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.664+0000 7f6c854b3640 1 -- 172.21.15.63:0/116680904 >> 172.21.15.63:0/116680904 conn(0x7f6c800b7ea0 msgr2=0x7f6c8014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:43.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.664+0000 7f6c854b3640 1 -- 172.21.15.63:0/116680904 shutdown_connections 2024-03-25T18:17:43.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.664+0000 7f6c854b3640 1 -- 172.21.15.63:0/116680904 wait complete. 2024-03-25T18:17:43.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:43.664+0000 7f6c854b3640 1 librados: shutdown 2024-03-25T18:17:43.676 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:50.482 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:50.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.601+0000 7f12e5d5d640 10 monclient: get_monmap_and_config 2024-03-25T18:17:50.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.601+0000 7f12e5d5d640 10 monclient: build_initial_monmap 2024-03-25T18:17:50.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.601+0000 7f12e5d5d640 10 monclient: monmap: 2024-03-25T18:17:50.602 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:17:50.602 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:50.602 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:17:50.602281+0000 2024-03-25T18:17:50.602 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:17:50.602281+0000 2024-03-25T18:17:50.602 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:50.602 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:50.602 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:17:50.603 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:17:50.603 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:17:50.603 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:50.604 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.601+0000 7f12e5d5d640 1 Processor -- start 2024-03-25T18:17:50.604 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.602+0000 7f12e5d5d640 1 -- start start 2024-03-25T18:17:50.604 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.602+0000 7f12e5d5d640 10 monclient: init 2024-03-25T18:17:50.604 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.602+0000 7f12e5d5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:50.604 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.602+0000 7f12e5d5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:50.604 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.602+0000 7f12e5d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f12e00a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:17:50.604 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.602+0000 7f12e5d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f12e013f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:17:50.604 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.602+0000 7f12e5d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f12e01430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:17:50.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.602+0000 7f12e5d5d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e0142520 con 0x7f12e00a98f0 2024-03-25T18:17:50.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.602+0000 7f12e5d5d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e00b28a0 con 0x7f12e013f710 2024-03-25T18:17:50.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.602+0000 7f12e5d5d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e00b2a30 con 0x7f12e01430c0 2024-03-25T18:17:50.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.602+0000 7f12e5d5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:50.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.602+0000 7f12e5d5d640 10 monclient(hunting): authenticate will time out at 7764.789094s 2024-03-25T18:17:50.613 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.612+0000 7f12deffd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f12e013f710 0x7f12e01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:56342/0 (socket says 172.21.15.63:56342) 2024-03-25T18:17:50.613 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.612+0000 7f12deffd640 1 -- 172.21.15.63:0/337972691 learned_addr learned my addr 172.21.15.63:0/337972691 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:17:50.622 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.621+0000 7f12de7fc640 1 -- 172.21.15.63:0/337972691 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441635627 0 0) 0x7f12e00b28a0 con 0x7f12e013f710 2024-03-25T18:17:50.622 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.621+0000 7f12de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:50.622 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.621+0000 7f12de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:50.622 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.621+0000 7f12de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:50.622 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.621+0000 7f12de7fc640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:17:50.622 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.621+0000 7f12de7fc640 1 -- 172.21.15.63:0/337972691 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c8003520 con 0x7f12e013f710 2024-03-25T18:17:50.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.625+0000 7f12de7fc640 1 -- 172.21.15.63:0/337972691 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4054728141 0 0) 0x7f12c8003520 con 0x7f12e013f710 2024-03-25T18:17:50.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.625+0000 7f12de7fc640 1 -- 172.21.15.63:0/337972691 >> v1:172.21.15.63:6790/0 conn(0x7f12e01430c0 legacy=0x7f12e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:50.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.625+0000 7f12de7fc640 1 -- 172.21.15.63:0/337972691 >> v1:172.21.15.63:6789/0 conn(0x7f12e00a98f0 legacy=0x7f12e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:50.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.625+0000 7f12de7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:50.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.625+0000 7f12de7fc640 1 monclient: found mon.noname-c 2024-03-25T18:17:50.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.625+0000 7f12de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:50.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.626+0000 7f12de7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:17:50.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.626+0000 7f12de7fc640 1 -- 172.21.15.63:0/337972691 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e00b44c0 con 0x7f12e013f710 2024-03-25T18:17:50.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.626+0000 7f12de7fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:50.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.626+0000 7f12de7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:50.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.626+0000 7f12de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:50.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.626+0000 7f12e5d5d640 5 monclient: authenticate success, global_id 4465 2024-03-25T18:17:50.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.626+0000 7f12e5d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:50.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.628+0000 7f12de7fc640 1 -- 172.21.15.63:0/337972691 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f12d4001690 con 0x7f12e013f710 2024-03-25T18:17:50.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.628+0000 7f12de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:50.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.628+0000 7f12de7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:50.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.628+0000 7f12de7fc640 10 monclient: dump: 2024-03-25T18:17:50.630 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:50.630 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:50.630 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:50.630 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:50.630 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:50.630 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:50.630 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:50.630 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:50.630 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:50.630 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:50.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.628+0000 7f12e5d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:50.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.632+0000 7f12de7fc640 1 -- 172.21.15.63:0/337972691 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12d4001920 con 0x7f12e013f710 2024-03-25T18:17:50.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.632+0000 7f12de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:50.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.632+0000 7f12e5d5d640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:50.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.632+0000 7f12e5d5d640 4 set_mon_vals no callback set 2024-03-25T18:17:50.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.633+0000 7f12de7fc640 1 -- 172.21.15.63:0/337972691 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f12d4001c30 con 0x7f12e013f710 2024-03-25T18:17:50.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.633+0000 7f12de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:50.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.633+0000 7f12de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:50.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.633+0000 7f12de7fc640 10 monclient: dump: 2024-03-25T18:17:50.634 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:50.634 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:50.634 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:50.634 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:50.634 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:50.634 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:50.635 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:50.635 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:50.635 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:50.635 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:50.636 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.633+0000 7f12e5d5d640 10 monclient: shutdown 2024-03-25T18:17:50.636 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.633+0000 7f12e5d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:50.636 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.633+0000 7f12e5d5d640 1 -- 172.21.15.63:0/337972691 >> v1:172.21.15.81:6789/0 conn(0x7f12e013f710 legacy=0x7f12e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:50.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.633+0000 7f12e5d5d640 1 -- 172.21.15.63:0/337972691 shutdown_connections 2024-03-25T18:17:50.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.633+0000 7f12e5d5d640 1 -- 172.21.15.63:0/337972691 >> 172.21.15.63:0/337972691 conn(0x7f12e00b7ea0 msgr2=0x7f12e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:50.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.633+0000 7f12e5d5d640 1 -- 172.21.15.63:0/337972691 shutdown_connections 2024-03-25T18:17:50.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.633+0000 7f12e5d5d640 1 -- 172.21.15.63:0/337972691 wait complete. 2024-03-25T18:17:50.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.634+0000 7f12e5d5d640 10 monclient: build_initial_monmap 2024-03-25T18:17:50.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.634+0000 7f12e5d5d640 10 monclient: monmap: 2024-03-25T18:17:50.637 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:17:50.637 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:50.637 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:17:50.637 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:17:50.637 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:50.637 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:17:50.637 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:17:50.638 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:17:50.638 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:17:50.638 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:50.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.635+0000 7f12e5d5d640 1 librados: starting msgr at 2024-03-25T18:17:50.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.635+0000 7f12e5d5d640 1 librados: starting objecter 2024-03-25T18:17:50.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.635+0000 7f12e5d5d640 1 Processor -- start 2024-03-25T18:17:50.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.635+0000 7f12e5d5d640 1 -- start start 2024-03-25T18:17:50.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.635+0000 7f12e5d5d640 1 librados: setting wanted keys 2024-03-25T18:17:50.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.635+0000 7f12e5d5d640 1 librados: calling monclient init 2024-03-25T18:17:50.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.635+0000 7f12e5d5d640 10 monclient: init 2024-03-25T18:17:50.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.636+0000 7f12e5d5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:50.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.636+0000 7f12e5d5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:50.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.636+0000 7f12e5d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f12e00a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:17:50.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.636+0000 7f12e5d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f12e013f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:17:50.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.636+0000 7f12e5d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f12e01430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:17:50.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.636+0000 7f12e5d5d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e0138c60 con 0x7f12e00a98f0 2024-03-25T18:17:50.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.636+0000 7f12e5d5d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e00abe30 con 0x7f12e01430c0 2024-03-25T18:17:50.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.636+0000 7f12e5d5d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e00ac110 con 0x7f12e013f710 2024-03-25T18:17:50.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.636+0000 7f12e5d5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:50.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.636+0000 7f12e5d5d640 10 monclient(hunting): authenticate will time out at 7764.822854s 2024-03-25T18:17:50.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.644+0000 7f12df7fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f12e00a98f0 0x7f12e0138370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:52262/0 (socket says 172.21.15.63:52262) 2024-03-25T18:17:50.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.644+0000 7f12df7fe640 1 -- 172.21.15.63:0/592124608 learned_addr learned my addr 172.21.15.63:0/592124608 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:17:50.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.649+0000 7f12c7fff640 10 client.?.objecter ms_handle_connect 0x7f12e00a98f0 2024-03-25T18:17:50.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.649+0000 7f12c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:50.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.656+0000 7f12c7fff640 10 client.?.objecter ms_handle_connect 0x7f12e01430c0 2024-03-25T18:17:50.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.656+0000 7f12c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:50.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.658+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364823485 0 0) 0x7f12e0138c60 con 0x7f12e00a98f0 2024-03-25T18:17:50.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.658+0000 7f12c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:50.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.658+0000 7f12c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:50.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.658+0000 7f12c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:50.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.658+0000 7f12c7fff640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:17:50.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.658+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12cc004eb0 con 0x7f12e00a98f0 2024-03-25T18:17:50.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.659+0000 7f12c7fff640 10 client.?.objecter ms_handle_connect 0x7f12e013f710 2024-03-25T18:17:50.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.659+0000 7f12c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:50.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.667+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300655930 0 0) 0x7f12e00abe30 con 0x7f12e01430c0 2024-03-25T18:17:50.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.667+0000 7f12c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:50.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.667+0000 7f12c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:50.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.667+0000 7f12c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:50.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.667+0000 7f12c7fff640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:17:50.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.667+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12e0138c60 con 0x7f12e01430c0 2024-03-25T18:17:50.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.667+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123665671 0 0) 0x7f12e00ac110 con 0x7f12e013f710 2024-03-25T18:17:50.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.667+0000 7f12c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:50.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.667+0000 7f12c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:50.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.667+0000 7f12c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:50.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.667+0000 7f12c7fff640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:17:50.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.667+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12e00abe30 con 0x7f12e013f710 2024-03-25T18:17:50.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.675+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4243607118 0 0) 0x7f12cc004eb0 con 0x7f12e00a98f0 2024-03-25T18:17:50.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.676+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12e00ac110 con 0x7f12e00a98f0 2024-03-25T18:17:50.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.676+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f12d00019d0 con 0x7f12e00a98f0 2024-03-25T18:17:50.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.676+0000 7f12c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:50.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.676+0000 7f12c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:50.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.676+0000 7f12c7fff640 10 monclient(hunting): dump: 2024-03-25T18:17:50.677 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:50.677 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:50.677 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:50.677 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:50.677 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:50.678 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:50.678 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:50.678 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:50.678 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:50.678 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:50.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.676+0000 7f12c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:50.682 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.681+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1642801739 0 0) 0x7f12e00abe30 con 0x7f12e013f710 2024-03-25T18:17:50.682 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.681+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12cc004eb0 con 0x7f12e013f710 2024-03-25T18:17:50.682 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.681+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f12d4001650 con 0x7f12e013f710 2024-03-25T18:17:50.682 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.681+0000 7f12c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:50.682 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.681+0000 7f12c7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:50.682 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.681+0000 7f12c7fff640 10 monclient(hunting): dump: 2024-03-25T18:17:50.683 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:50.683 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:50.683 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:50.683 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:50.683 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:50.683 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:50.683 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:50.683 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:50.683 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:50.683 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:50.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.693+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3588036782 0 0) 0x7f12e0138c60 con 0x7f12e01430c0 2024-03-25T18:17:50.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.693+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12e00abe30 con 0x7f12e01430c0 2024-03-25T18:17:50.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.693+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f12d8001b10 con 0x7f12e01430c0 2024-03-25T18:17:50.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.693+0000 7f12c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:50.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.693+0000 7f12c7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:50.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.693+0000 7f12c7fff640 10 monclient(hunting): dump: 2024-03-25T18:17:50.695 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:50.695 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:50.695 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:50.695 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:50.695 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:50.695 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:50.695 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:50.696 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:50.696 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:50.696 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:50.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1229713621 0 0) 0x7f12cc004eb0 con 0x7f12e013f710 2024-03-25T18:17:50.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 >> v1:172.21.15.81:6789/0 conn(0x7f12e01430c0 legacy=0x7f12e00ac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:50.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 >> v1:172.21.15.63:6789/0 conn(0x7f12e00a98f0 legacy=0x7f12e0138370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:50.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12c7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:50.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12c7fff640 1 monclient: found mon.c 2024-03-25T18:17:50.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:50.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12c7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:17:50.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e00ac3f0 con 0x7f12e013f710 2024-03-25T18:17:50.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12c7fff640 10 monclient: _finish_auth 0 2024-03-25T18:17:50.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12c7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:50.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:50.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12e5d5d640 5 monclient: authenticate success, global_id 4451 2024-03-25T18:17:50.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12e5d5d640 10 monclient: _renew_subs 2024-03-25T18:17:50.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12e5d5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:17:50.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12e5d5d640 1 -- 172.21.15.63:0/592124608 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e0136c40 con 0x7f12e013f710 2024-03-25T18:17:50.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12e5d5d640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:50.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12e5d5d640 10 monclient: _renew_subs 2024-03-25T18:17:50.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12e5d5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:17:50.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12e5d5d640 1 -- 172.21.15.63:0/592124608 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e01e38a0 con 0x7f12e013f710 2024-03-25T18:17:50.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.697+0000 7f12e5d5d640 1 librados: init done 2024-03-25T18:17:50.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.698+0000 7f12e5d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:50.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.699+0000 7f12e5d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:50.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.699+0000 7f12e5d5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:17:50.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.699+0000 7f12e5d5d640 1 -- 172.21.15.63:0/592124608 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e00b3ef0 con 0x7f12e013f710 2024-03-25T18:17:50.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.996+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12d4004a90 con 0x7f12e013f710 2024-03-25T18:17:50.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.996+0000 7f12c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:50.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.996+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f12d4004da0 con 0x7f12e013f710 2024-03-25T18:17:50.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.996+0000 7f12c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:50.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.996+0000 7f12c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:50.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.996+0000 7f12c7fff640 10 monclient: dump: 2024-03-25T18:17:50.997 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:50.997 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:50.998 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:50.998 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:50.998 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:50.998 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:50.998 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:50.998 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:50.998 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:50.998 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:50.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.996+0000 7f12ddffb640 4 set_mon_vals no callback set 2024-03-25T18:17:50.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.996+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f12d401ad70 con 0x7f12e013f710 2024-03-25T18:17:50.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.997+0000 7f12df7fe640 10 monclient: get_auth_request con 0x7f12cc06e110 auth_method 0 2024-03-25T18:17:50.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:50.997+0000 7f12c7fff640 10 client.4451.objecter ms_handle_connect 0x7f12cc06e110 2024-03-25T18:17:51.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.003+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f12d40547b0 con 0x7f12e013f710 2024-03-25T18:17:51.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.003+0000 7f12c7fff640 10 client.4451.objecter ms_dispatch 0x7f12e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:51.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.003+0000 7f12c7fff640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:51.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.003+0000 7f12c7fff640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:51.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.003+0000 7f12c7fff640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-25T18:17:51.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.003+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f12d40502c0 con 0x7f12e013f710 2024-03-25T18:17:51.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.003+0000 7f12c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:51.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.003+0000 7f12c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:51.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.257+0000 7f12e5d5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:51.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.257+0000 7f12e5d5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:51.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.257+0000 7f12e5d5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:17:51.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.257+0000 7f12e5d5d640 1 -- 172.21.15.63:0/592124608 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12e0138e10 con 0x7f12e013f710 2024-03-25T18:17:51.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.385+0000 7f12c7fff640 1 -- 172.21.15.63:0/592124608 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f12e0138e10 con 0x7f12e013f710 2024-03-25T18:17:51.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.385+0000 7f12c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:51.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.385+0000 7f12c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:51.387 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:17:51.387 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:51.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.388+0000 7f12e5d5d640 10 librados: watch_flush enter 2024-03-25T18:17:51.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.388+0000 7f12e5d5d640 10 librados: watch_flush exit 2024-03-25T18:17:51.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.388+0000 7f12e5d5d640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-25T18:17:51.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.388+0000 7f12e5d5d640 10 client.4451.objecter successfully canceled tick 2024-03-25T18:17:51.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.388+0000 7f12e5d5d640 1 -- 172.21.15.63:0/592124608 >> v1:172.21.15.81:6800/3640340873 conn(0x7f12cc06e110 legacy=0x7f12cc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:51.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.388+0000 7f12e5d5d640 10 monclient: shutdown 2024-03-25T18:17:51.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.388+0000 7f12e5d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:51.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.388+0000 7f12e5d5d640 1 -- 172.21.15.63:0/592124608 >> v1:172.21.15.63:6790/0 conn(0x7f12e013f710 legacy=0x7f12e01390f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:51.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.388+0000 7f12e5d5d640 1 -- 172.21.15.63:0/592124608 shutdown_connections 2024-03-25T18:17:51.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.388+0000 7f12e5d5d640 1 -- 172.21.15.63:0/592124608 >> 172.21.15.63:0/592124608 conn(0x7f12e00b7ea0 msgr2=0x7f12e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:51.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.388+0000 7f12e5d5d640 1 -- 172.21.15.63:0/592124608 shutdown_connections 2024-03-25T18:17:51.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.388+0000 7f12e5d5d640 1 -- 172.21.15.63:0/592124608 wait complete. 2024-03-25T18:17:51.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:51.388+0000 7f12e5d5d640 1 librados: shutdown 2024-03-25T18:17:51.402 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:58.209 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:58.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.322+0000 7f08dd2a8640 10 monclient: get_monmap_and_config 2024-03-25T18:17:58.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.322+0000 7f08dd2a8640 10 monclient: build_initial_monmap 2024-03-25T18:17:58.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.322+0000 7f08dd2a8640 10 monclient: monmap: 2024-03-25T18:17:58.324 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:17:58.324 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:58.324 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:17:58.323892+0000 2024-03-25T18:17:58.324 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:17:58.323892+0000 2024-03-25T18:17:58.324 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:58.324 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:58.324 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:17:58.324 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:17:58.325 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:17:58.325 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:58.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.323+0000 7f08dd2a8640 1 Processor -- start 2024-03-25T18:17:58.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.323+0000 7f08dd2a8640 1 -- start start 2024-03-25T18:17:58.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.323+0000 7f08dd2a8640 10 monclient: init 2024-03-25T18:17:58.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.324+0000 7f08dd2a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:58.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.324+0000 7f08dd2a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:58.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.324+0000 7f08dd2a8640 10 monclient(hunting): picked mon.noname-c con 0x7f08d80705b0 addr v1:172.21.15.81:6789/0 2024-03-25T18:17:58.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.324+0000 7f08dd2a8640 10 monclient(hunting): picked mon.noname-a con 0x7f08d81402d0 addr v1:172.21.15.63:6789/0 2024-03-25T18:17:58.327 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.324+0000 7f08dd2a8640 10 monclient(hunting): picked mon.noname-b con 0x7f08d8149ae0 addr v1:172.21.15.63:6790/0 2024-03-25T18:17:58.327 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.324+0000 7f08dd2a8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d80718c0 con 0x7f08d81402d0 2024-03-25T18:17:58.327 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.324+0000 7f08dd2a8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d80714c0 con 0x7f08d80705b0 2024-03-25T18:17:58.327 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.324+0000 7f08dd2a8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d8072b50 con 0x7f08d8149ae0 2024-03-25T18:17:58.327 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.324+0000 7f08dd2a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:58.327 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.324+0000 7f08dd2a8640 10 monclient(hunting): authenticate will time out at 7772.510875s 2024-03-25T18:17:58.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.335+0000 7f08d6ffd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f08d80705b0 0x7f08d813d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:33122/0 (socket says 172.21.15.63:33122) 2024-03-25T18:17:58.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.335+0000 7f08d6ffd640 1 -- 172.21.15.63:0/3660706427 learned_addr learned my addr 172.21.15.63:0/3660706427 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:17:58.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.341+0000 7f08d5ffb640 1 -- 172.21.15.63:0/3660706427 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575905724 0 0) 0x7f08d80714c0 con 0x7f08d80705b0 2024-03-25T18:17:58.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.341+0000 7f08d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:58.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.341+0000 7f08d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:58.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.341+0000 7f08d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:58.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.341+0000 7f08d5ffb640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:17:58.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.341+0000 7f08d5ffb640 1 -- 172.21.15.63:0/3660706427 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c0003520 con 0x7f08d80705b0 2024-03-25T18:17:58.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.346+0000 7f08d5ffb640 1 -- 172.21.15.63:0/3660706427 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2691435503 0 0) 0x7f08c0003520 con 0x7f08d80705b0 2024-03-25T18:17:58.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.346+0000 7f08d5ffb640 1 -- 172.21.15.63:0/3660706427 >> v1:172.21.15.63:6790/0 conn(0x7f08d8149ae0 legacy=0x7f08d814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:58.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.346+0000 7f08d5ffb640 1 -- 172.21.15.63:0/3660706427 >> v1:172.21.15.63:6789/0 conn(0x7f08d81402d0 legacy=0x7f08d8140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:58.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.346+0000 7f08d5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:58.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.346+0000 7f08d5ffb640 1 monclient: found mon.noname-c 2024-03-25T18:17:58.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.346+0000 7f08d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:58.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.346+0000 7f08d5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:17:58.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.346+0000 7f08d5ffb640 1 -- 172.21.15.63:0/3660706427 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d813dd10 con 0x7f08d80705b0 2024-03-25T18:17:58.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.346+0000 7f08d5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:58.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.346+0000 7f08d5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:58.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.346+0000 7f08d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:58.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.346+0000 7f08dd2a8640 5 monclient: authenticate success, global_id 4471 2024-03-25T18:17:58.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.346+0000 7f08dd2a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:58.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.350+0000 7f08d5ffb640 1 -- 172.21.15.63:0/3660706427 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f08c8001690 con 0x7f08d80705b0 2024-03-25T18:17:58.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.350+0000 7f08d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:58.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.350+0000 7f08d5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:58.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.350+0000 7f08d5ffb640 10 monclient: dump: 2024-03-25T18:17:58.352 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:58.352 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:58.352 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:58.352 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:58.352 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:58.352 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:58.352 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:58.352 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:58.352 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:58.352 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:58.352 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.350+0000 7f08dd2a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:58.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.352+0000 7f08d5ffb640 1 -- 172.21.15.63:0/3660706427 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08c8001920 con 0x7f08d80705b0 2024-03-25T18:17:58.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.352+0000 7f08d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:58.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.352+0000 7f08d5ffb640 1 -- 172.21.15.63:0/3660706427 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f08c8001c30 con 0x7f08d80705b0 2024-03-25T18:17:58.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.352+0000 7f08dd2a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:58.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.352+0000 7f08dd2a8640 4 set_mon_vals no callback set 2024-03-25T18:17:58.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.352+0000 7f08d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:58.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.353+0000 7f08d5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:58.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.353+0000 7f08d5ffb640 10 monclient: dump: 2024-03-25T18:17:58.355 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:58.355 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:58.355 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:58.355 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:58.355 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:58.355 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:58.355 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:58.355 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:58.355 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:58.355 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:58.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.353+0000 7f08dd2a8640 10 monclient: shutdown 2024-03-25T18:17:58.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.353+0000 7f08dd2a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:58.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.353+0000 7f08dd2a8640 1 -- 172.21.15.63:0/3660706427 >> v1:172.21.15.81:6789/0 conn(0x7f08d80705b0 legacy=0x7f08d813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:58.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.353+0000 7f08dd2a8640 1 -- 172.21.15.63:0/3660706427 shutdown_connections 2024-03-25T18:17:58.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.353+0000 7f08dd2a8640 1 -- 172.21.15.63:0/3660706427 >> 172.21.15.63:0/3660706427 conn(0x7f08d80b5d10 msgr2=0x7f08d80b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:58.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.353+0000 7f08dd2a8640 1 -- 172.21.15.63:0/3660706427 shutdown_connections 2024-03-25T18:17:58.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.354+0000 7f08dd2a8640 1 -- 172.21.15.63:0/3660706427 wait complete. 2024-03-25T18:17:58.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.354+0000 7f08dd2a8640 10 monclient: build_initial_monmap 2024-03-25T18:17:58.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.354+0000 7f08dd2a8640 10 monclient: monmap: 2024-03-25T18:17:58.358 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:17:58.358 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:58.358 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:17:58.358 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:17:58.359 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:58.359 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:17:58.359 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:17:58.359 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:17:58.359 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:17:58.359 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:58.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.355+0000 7f08dd2a8640 1 librados: starting msgr at 2024-03-25T18:17:58.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.355+0000 7f08dd2a8640 1 librados: starting objecter 2024-03-25T18:17:58.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.355+0000 7f08dd2a8640 1 Processor -- start 2024-03-25T18:17:58.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.355+0000 7f08dd2a8640 1 -- start start 2024-03-25T18:17:58.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.355+0000 7f08dd2a8640 1 librados: setting wanted keys 2024-03-25T18:17:58.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.355+0000 7f08dd2a8640 1 librados: calling monclient init 2024-03-25T18:17:58.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.355+0000 7f08dd2a8640 10 monclient: init 2024-03-25T18:17:58.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.356+0000 7f08dd2a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:58.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.356+0000 7f08dd2a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:58.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.356+0000 7f08dd2a8640 10 monclient(hunting): picked mon.noname-c con 0x7f08d80705b0 addr v1:172.21.15.63:6790/0 2024-03-25T18:17:58.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.356+0000 7f08dd2a8640 10 monclient(hunting): picked mon.noname-a con 0x7f08d81402d0 addr v1:172.21.15.63:6789/0 2024-03-25T18:17:58.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.356+0000 7f08dd2a8640 10 monclient(hunting): picked mon.noname-b con 0x7f08d8149ae0 addr v1:172.21.15.81:6789/0 2024-03-25T18:17:58.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.356+0000 7f08dd2a8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d81d7510 con 0x7f08d81402d0 2024-03-25T18:17:58.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.356+0000 7f08dd2a8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d81d83e0 con 0x7f08d8149ae0 2024-03-25T18:17:58.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.356+0000 7f08dd2a8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d81d77b0 con 0x7f08d80705b0 2024-03-25T18:17:58.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.356+0000 7f08dd2a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:58.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.356+0000 7f08dd2a8640 10 monclient(hunting): authenticate will time out at 7772.542544s 2024-03-25T18:17:58.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.362+0000 7f08d6ffd640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f08d80705b0 0x7f08d81d7cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:55558/0 (socket says 172.21.15.63:55558) 2024-03-25T18:17:58.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.362+0000 7f08d6ffd640 1 -- 172.21.15.63:0/210999187 learned_addr learned my addr 172.21.15.63:0/210999187 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:17:58.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.364+0000 7f08bf7fe640 10 client.?.objecter ms_handle_connect 0x7f08d80705b0 2024-03-25T18:17:58.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.364+0000 7f08bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:58.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.365+0000 7f08bf7fe640 10 client.?.objecter ms_handle_connect 0x7f08d81402d0 2024-03-25T18:17:58.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.365+0000 7f08bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:58.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.366+0000 7f08bf7fe640 10 client.?.objecter ms_handle_connect 0x7f08d8149ae0 2024-03-25T18:17:58.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.366+0000 7f08bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:58.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.375+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120365100 0 0) 0x7f08d81d83e0 con 0x7f08d8149ae0 2024-03-25T18:17:58.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.375+0000 7f08bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:58.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.375+0000 7f08bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:58.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.375+0000 7f08bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:58.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.375+0000 7f08bf7fe640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:17:58.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.375+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c4004eb0 con 0x7f08d8149ae0 2024-03-25T18:17:58.380 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.379+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979795260 0 0) 0x7f08d81d77b0 con 0x7f08d80705b0 2024-03-25T18:17:58.380 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.379+0000 7f08bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:58.380 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.379+0000 7f08bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:58.380 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.379+0000 7f08bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:58.380 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.379+0000 7f08bf7fe640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:17:58.381 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.379+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08d81d83e0 con 0x7f08d80705b0 2024-03-25T18:17:58.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.382+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544780437 0 0) 0x7f08d81d7510 con 0x7f08d81402d0 2024-03-25T18:17:58.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.382+0000 7f08bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:58.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.382+0000 7f08bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:58.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.382+0000 7f08bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:58.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.382+0000 7f08bf7fe640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:17:58.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.382+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08d81d77b0 con 0x7f08d81402d0 2024-03-25T18:17:58.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.386+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2636372174 0 0) 0x7f08c4004eb0 con 0x7f08d8149ae0 2024-03-25T18:17:58.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.386+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08d81d7510 con 0x7f08d8149ae0 2024-03-25T18:17:58.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.386+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f08d0001b90 con 0x7f08d8149ae0 2024-03-25T18:17:58.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.386+0000 7f08bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:58.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.386+0000 7f08bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:58.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.386+0000 7f08bf7fe640 10 monclient(hunting): dump: 2024-03-25T18:17:58.388 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:58.388 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:58.388 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:58.388 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:58.388 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:58.388 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:58.388 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:58.388 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:58.389 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:58.389 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:58.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.386+0000 7f08bf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:58.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.386+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2093688723 0 0) 0x7f08d81d83e0 con 0x7f08d80705b0 2024-03-25T18:17:58.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.386+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08c4004eb0 con 0x7f08d80705b0 2024-03-25T18:17:58.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.386+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f08c8001650 con 0x7f08d80705b0 2024-03-25T18:17:58.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.386+0000 7f08bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:58.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.386+0000 7f08bf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:58.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.386+0000 7f08bf7fe640 10 monclient(hunting): dump: 2024-03-25T18:17:58.389 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:58.389 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:58.389 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:58.390 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:58.390 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:58.390 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:58.390 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:58.390 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:58.390 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:58.390 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:58.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 919177422 0 0) 0x7f08d81d7510 con 0x7f08d8149ae0 2024-03-25T18:17:58.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 >> v1:172.21.15.63:6790/0 conn(0x7f08d80705b0 legacy=0x7f08d81d7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:58.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 >> v1:172.21.15.63:6789/0 conn(0x7f08d81402d0 legacy=0x7f08d80aa620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:58.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08bf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:58.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08bf7fe640 1 monclient: found mon.b 2024-03-25T18:17:58.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:58.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08bf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:58.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d80aad30 con 0x7f08d8149ae0 2024-03-25T18:17:58.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08bf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:58.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08bf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:58.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:58.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08dd2a8640 5 monclient: authenticate success, global_id 4474 2024-03-25T18:17:58.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08dd2a8640 10 monclient: _renew_subs 2024-03-25T18:17:58.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08dd2a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:58.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08dd2a8640 1 -- 172.21.15.63:0/210999187 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d8141dd0 con 0x7f08d8149ae0 2024-03-25T18:17:58.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08dd2a8640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:58.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08dd2a8640 10 monclient: _renew_subs 2024-03-25T18:17:58.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08dd2a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:58.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08dd2a8640 1 -- 172.21.15.63:0/210999187 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d80a9880 con 0x7f08d8149ae0 2024-03-25T18:17:58.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.390+0000 7f08dd2a8640 1 librados: init done 2024-03-25T18:17:58.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.391+0000 7f08dd2a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:58.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.391+0000 7f08dd2a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:58.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.391+0000 7f08dd2a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:58.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.391+0000 7f08dd2a8640 1 -- 172.21.15.63:0/210999187 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d81d7940 con 0x7f08d8149ae0 2024-03-25T18:17:58.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.667+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08d0002080 con 0x7f08d8149ae0 2024-03-25T18:17:58.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.667+0000 7f08bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:58.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.667+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f08d0003b10 con 0x7f08d8149ae0 2024-03-25T18:17:58.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.667+0000 7f08bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:58.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.667+0000 7f08bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:58.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.667+0000 7f08bf7fe640 10 monclient: dump: 2024-03-25T18:17:58.669 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:17:58.669 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:17:58.669 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:58.669 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:17:58.669 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:58.669 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:17:58.669 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:17:58.669 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:17:58.670 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:17:58.670 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:17:58.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.667+0000 7f08d57fa640 4 set_mon_vals no callback set 2024-03-25T18:17:58.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.668+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f08d0003d30 con 0x7f08d8149ae0 2024-03-25T18:17:58.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.668+0000 7f08d6ffd640 10 monclient: get_auth_request con 0x7f08c406df20 auth_method 0 2024-03-25T18:17:58.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.669+0000 7f08bf7fe640 10 client.4474.objecter ms_handle_connect 0x7f08c406df20 2024-03-25T18:17:58.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.672+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f08d0053800 con 0x7f08d8149ae0 2024-03-25T18:17:58.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.672+0000 7f08bf7fe640 10 client.4474.objecter ms_dispatch 0x7f08d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:58.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.672+0000 7f08bf7fe640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:58.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.672+0000 7f08bf7fe640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:58.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.672+0000 7f08bf7fe640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-25T18:17:58.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.673+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08d004f310 con 0x7f08d8149ae0 2024-03-25T18:17:58.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.673+0000 7f08bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:58.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.673+0000 7f08bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:58.905 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.904+0000 7f08dd2a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:58.905 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.904+0000 7f08dd2a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:58.905 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.904+0000 7f08dd2a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:17:58.905 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.904+0000 7f08dd2a8640 1 -- 172.21.15.63:0/210999187 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08d80627b0 con 0x7f08d8149ae0 2024-03-25T18:17:58.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.925+0000 7f08bf7fe640 1 -- 172.21.15.63:0/210999187 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f08d0051240 con 0x7f08d8149ae0 2024-03-25T18:17:58.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.925+0000 7f08bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:58.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.925+0000 7f08bf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:58.926 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:17:58.926 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.927+0000 7f08dd2a8640 10 librados: watch_flush enter 2024-03-25T18:17:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.927+0000 7f08dd2a8640 10 librados: watch_flush exit 2024-03-25T18:17:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.927+0000 7f08dd2a8640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-25T18:17:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.927+0000 7f08dd2a8640 10 client.4474.objecter successfully canceled tick 2024-03-25T18:17:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.927+0000 7f08dd2a8640 1 -- 172.21.15.63:0/210999187 >> v1:172.21.15.81:6800/3640340873 conn(0x7f08c406df20 legacy=0x7f08c40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.927+0000 7f08dd2a8640 10 monclient: shutdown 2024-03-25T18:17:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.927+0000 7f08dd2a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.927+0000 7f08dd2a8640 1 -- 172.21.15.63:0/210999187 >> v1:172.21.15.81:6789/0 conn(0x7f08d8149ae0 legacy=0x7f08d8139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.927+0000 7f08dd2a8640 1 -- 172.21.15.63:0/210999187 shutdown_connections 2024-03-25T18:17:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.927+0000 7f08dd2a8640 1 -- 172.21.15.63:0/210999187 >> 172.21.15.63:0/210999187 conn(0x7f08d80b5d10 msgr2=0x7f08d80b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.927+0000 7f08dd2a8640 1 -- 172.21.15.63:0/210999187 shutdown_connections 2024-03-25T18:17:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.927+0000 7f08dd2a8640 1 -- 172.21.15.63:0/210999187 wait complete. 2024-03-25T18:17:58.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:17:58.927+0000 7f08dd2a8640 1 librados: shutdown 2024-03-25T18:17:58.941 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:58.986 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:40.679 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:59.048 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:40.740 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:59.094 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:40.786 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:59.102 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:05:40.762 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:59.105 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:05:40.766 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:59.118 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:40.810 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:59.121 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:40.814 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:59.147 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:05:40.808 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:59.156 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:40.849 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:05.747 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:05.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.863+0000 7f6a954d3640 10 monclient: get_monmap_and_config 2024-03-25T18:18:05.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.863+0000 7f6a954d3640 10 monclient: build_initial_monmap 2024-03-25T18:18:05.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.863+0000 7f6a954d3640 10 monclient: monmap: 2024-03-25T18:18:05.865 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:18:05.865 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:05.865 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:18:05.865020+0000 2024-03-25T18:18:05.865 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:18:05.865020+0000 2024-03-25T18:18:05.865 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:05.865 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:05.865 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:18:05.865 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:18:05.865 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:18:05.865 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:05.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.864+0000 7f6a954d3640 1 Processor -- start 2024-03-25T18:18:05.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.864+0000 7f6a954d3640 1 -- start start 2024-03-25T18:18:05.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.864+0000 7f6a954d3640 10 monclient: init 2024-03-25T18:18:05.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.865+0000 7f6a954d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:05.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.865+0000 7f6a954d3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:05.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.865+0000 7f6a954d3640 10 monclient(hunting): picked mon.noname-a con 0x7f6a900a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:18:05.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.865+0000 7f6a954d3640 10 monclient(hunting): picked mon.noname-c con 0x7f6a9013f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:18:05.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.865+0000 7f6a954d3640 10 monclient(hunting): picked mon.noname-b con 0x7f6a901430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:18:05.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.865+0000 7f6a954d3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a90142520 con 0x7f6a900a98f0 2024-03-25T18:18:05.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.865+0000 7f6a954d3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a900b28a0 con 0x7f6a9013f710 2024-03-25T18:18:05.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.865+0000 7f6a954d3640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a900b2a30 con 0x7f6a901430c0 2024-03-25T18:18:05.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.865+0000 7f6a954d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:05.868 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.865+0000 7f6a954d3640 10 monclient(hunting): authenticate will time out at 7780.051692s 2024-03-25T18:18:05.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.873+0000 7f6a8f7fe640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f6a901430c0 0x7f6a9014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:52090/0 (socket says 172.21.15.63:52090) 2024-03-25T18:18:05.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.873+0000 7f6a8f7fe640 1 -- 172.21.15.63:0/4046540245 learned_addr learned my addr 172.21.15.63:0/4046540245 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:18:05.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.877+0000 7f6a8e7fc640 1 -- 172.21.15.63:0/4046540245 >> v1:172.21.15.81:6789/0 conn(0x7f6a9013f710 legacy=0x7f6a901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T18:18:05.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.877+0000 7f6a8e7fc640 1 -- 172.21.15.63:0/4046540245 >> v1:172.21.15.81:6789/0 conn(0x7f6a9013f710 legacy=0x7f6a901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:18:05.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.877+0000 7f6a8e7fc640 1 --1- 172.21.15.63:0/4046540245 >> v1:172.21.15.81:6789/0 conn(0x7f6a9013f710 0x7f6a901429b0 :-1 s=OPENED pgs=38814 cs=1 l=1).handle_message read tag failed 2024-03-25T18:18:05.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.877+0000 7f6a8e7fc640 1 --1- 172.21.15.63:0/4046540245 >> v1:172.21.15.81:6789/0 conn(0x7f6a9013f710 0x7f6a901429b0 :-1 s=OPENED pgs=38814 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:18:05.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.877+0000 7f6a8dffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.81:6789/0 2024-03-25T18:18:05.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.888+0000 7f6a8dffb640 1 -- 172.21.15.63:0/4046540245 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304872957 0 0) 0x7f6a900b2a30 con 0x7f6a901430c0 2024-03-25T18:18:05.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.888+0000 7f6a8dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:05.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.888+0000 7f6a8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:05.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.888+0000 7f6a8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:05.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.888+0000 7f6a8dffb640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:18:05.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.888+0000 7f6a8dffb640 1 -- 172.21.15.63:0/4046540245 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a78003520 con 0x7f6a901430c0 2024-03-25T18:18:05.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.890+0000 7f6a8dffb640 1 -- 172.21.15.63:0/4046540245 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543714581 0 0) 0x7f6a90142520 con 0x7f6a900a98f0 2024-03-25T18:18:05.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.890+0000 7f6a8dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:05.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.890+0000 7f6a8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:05.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.890+0000 7f6a8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:05.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.890+0000 7f6a8dffb640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:18:05.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.890+0000 7f6a8dffb640 1 -- 172.21.15.63:0/4046540245 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a900b2a30 con 0x7f6a900a98f0 2024-03-25T18:18:05.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a8dffb640 1 -- 172.21.15.63:0/4046540245 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1521864165 0 0) 0x7f6a78003520 con 0x7f6a901430c0 2024-03-25T18:18:05.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a8dffb640 1 -- 172.21.15.63:0/4046540245 >> v1:172.21.15.81:6789/0 conn(0x7f6a9013f710 legacy=0x7f6a901429b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:18:05.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a8dffb640 1 -- 172.21.15.63:0/4046540245 >> v1:172.21.15.63:6789/0 conn(0x7f6a900a98f0 legacy=0x7f6a900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:05.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a8dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:05.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a8dffb640 1 monclient: found mon.noname-b 2024-03-25T18:18:05.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:05.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a8dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:18:05.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a8dffb640 1 -- 172.21.15.63:0/4046540245 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a900b44c0 con 0x7f6a901430c0 2024-03-25T18:18:05.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a8dffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:05.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a8dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:05.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:05.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a8dffb640 1 -- 172.21.15.63:0/4046540245 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6a88001a10 con 0x7f6a901430c0 2024-03-25T18:18:05.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a954d3640 5 monclient: authenticate success, global_id 4460 2024-03-25T18:18:05.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a954d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:05.900 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:05.900 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a8dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:05.900 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a8dffb640 10 monclient: dump: 2024-03-25T18:18:05.900 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:05.900 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:05.900 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:05.900 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:05.900 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:05.900 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:05.900 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:05.900 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:05.900 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:05.900 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:05.900 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.897+0000 7f6a954d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:05.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.903+0000 7f6a8dffb640 1 -- 172.21.15.63:0/4046540245 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a88001b70 con 0x7f6a901430c0 2024-03-25T18:18:05.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.903+0000 7f6a8dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:05.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.903+0000 7f6a8dffb640 1 -- 172.21.15.63:0/4046540245 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6a88001f80 con 0x7f6a901430c0 2024-03-25T18:18:05.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.903+0000 7f6a8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:05.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.903+0000 7f6a8dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:05.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.903+0000 7f6a8dffb640 10 monclient: dump: 2024-03-25T18:18:05.904 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:05.905 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:05.905 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:05.905 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:05.905 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:05.905 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:05.905 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:05.905 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:05.905 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:05.905 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:05.905 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.903+0000 7f6a954d3640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:05.905 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.903+0000 7f6a954d3640 4 set_mon_vals no callback set 2024-03-25T18:18:05.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.904+0000 7f6a954d3640 10 monclient: shutdown 2024-03-25T18:18:05.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.904+0000 7f6a954d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:05.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.904+0000 7f6a954d3640 1 -- 172.21.15.63:0/4046540245 >> v1:172.21.15.63:6790/0 conn(0x7f6a901430c0 legacy=0x7f6a9014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:05.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.904+0000 7f6a954d3640 1 -- 172.21.15.63:0/4046540245 shutdown_connections 2024-03-25T18:18:05.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.904+0000 7f6a954d3640 1 -- 172.21.15.63:0/4046540245 >> 172.21.15.63:0/4046540245 conn(0x7f6a900b7ea0 msgr2=0x7f6a900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:05.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.904+0000 7f6a954d3640 1 -- 172.21.15.63:0/4046540245 shutdown_connections 2024-03-25T18:18:05.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.904+0000 7f6a954d3640 1 -- 172.21.15.63:0/4046540245 wait complete. 2024-03-25T18:18:05.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.905+0000 7f6a954d3640 10 monclient: build_initial_monmap 2024-03-25T18:18:05.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.905+0000 7f6a954d3640 10 monclient: monmap: 2024-03-25T18:18:05.907 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:18:05.907 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:05.907 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:18:05.907 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:18:05.908 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:05.908 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:18:05.908 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:18:05.908 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:18:05.908 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:18:05.908 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:05.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.905+0000 7f6a954d3640 1 librados: starting msgr at 2024-03-25T18:18:05.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.905+0000 7f6a954d3640 1 librados: starting objecter 2024-03-25T18:18:05.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.906+0000 7f6a954d3640 1 Processor -- start 2024-03-25T18:18:05.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.906+0000 7f6a954d3640 1 -- start start 2024-03-25T18:18:05.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.906+0000 7f6a954d3640 1 librados: setting wanted keys 2024-03-25T18:18:05.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.906+0000 7f6a954d3640 1 librados: calling monclient init 2024-03-25T18:18:05.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.906+0000 7f6a954d3640 10 monclient: init 2024-03-25T18:18:05.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.906+0000 7f6a954d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:05.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.906+0000 7f6a954d3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:05.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.906+0000 7f6a954d3640 10 monclient(hunting): picked mon.noname-a con 0x7f6a900a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:18:05.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.906+0000 7f6a954d3640 10 monclient(hunting): picked mon.noname-b con 0x7f6a9013f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:18:05.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.906+0000 7f6a954d3640 10 monclient(hunting): picked mon.noname-c con 0x7f6a901430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:18:05.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.906+0000 7f6a954d3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a90139410 con 0x7f6a900a98f0 2024-03-25T18:18:05.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.906+0000 7f6a954d3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a900ac240 con 0x7f6a9013f710 2024-03-25T18:18:05.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.906+0000 7f6a954d3640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a900ac520 con 0x7f6a901430c0 2024-03-25T18:18:05.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.906+0000 7f6a954d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:05.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.906+0000 7f6a954d3640 10 monclient(hunting): authenticate will time out at 7780.093119s 2024-03-25T18:18:05.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.914+0000 7f6a8e7fc640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f6a9013f710 0x7f6a90135880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:49298/0 (socket says 172.21.15.63:49298) 2024-03-25T18:18:05.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.914+0000 7f6a8e7fc640 1 -- 172.21.15.63:0/2811392052 learned_addr learned my addr 172.21.15.63:0/2811392052 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:18:05.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.915+0000 7f6a777fe640 10 client.?.objecter ms_handle_connect 0x7f6a9013f710 2024-03-25T18:18:05.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.915+0000 7f6a777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:05.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.916+0000 7f6a777fe640 10 client.?.objecter ms_handle_connect 0x7f6a900a98f0 2024-03-25T18:18:05.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.916+0000 7f6a777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:05.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.920+0000 7f6a777fe640 1 -- 172.21.15.63:0/2811392052 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225110527 0 0) 0x7f6a900ac240 con 0x7f6a9013f710 2024-03-25T18:18:05.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.920+0000 7f6a777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:05.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.920+0000 7f6a777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:05.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.920+0000 7f6a777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:05.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.920+0000 7f6a777fe640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:18:05.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.920+0000 7f6a777fe640 1 -- 172.21.15.63:0/2811392052 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a7c004eb0 con 0x7f6a9013f710 2024-03-25T18:18:05.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.927+0000 7f6a777fe640 1 -- 172.21.15.63:0/2811392052 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3219613272 0 0) 0x7f6a7c004eb0 con 0x7f6a9013f710 2024-03-25T18:18:05.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.927+0000 7f6a777fe640 1 -- 172.21.15.63:0/2811392052 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a900ac240 con 0x7f6a9013f710 2024-03-25T18:18:05.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.927+0000 7f6a777fe640 1 -- 172.21.15.63:0/2811392052 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6a84001a20 con 0x7f6a9013f710 2024-03-25T18:18:05.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.927+0000 7f6a777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:05.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.927+0000 7f6a777fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:05.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.927+0000 7f6a777fe640 10 monclient(hunting): dump: 2024-03-25T18:18:05.928 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:05.928 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:05.929 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:05.929 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:05.929 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:05.929 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:05.929 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:05.929 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:05.929 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:05.929 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:05.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.927+0000 7f6a777fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:05.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.932+0000 7f6a777fe640 1 -- 172.21.15.63:0/2811392052 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 854859457 0 0) 0x7f6a900ac240 con 0x7f6a9013f710 2024-03-25T18:18:05.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.932+0000 7f6a777fe640 1 -- 172.21.15.63:0/2811392052 >> v1:172.21.15.63:6790/0 conn(0x7f6a901430c0 legacy=0x7f6a901e2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:05.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.932+0000 7f6a777fe640 1 -- 172.21.15.63:0/2811392052 >> v1:172.21.15.63:6789/0 conn(0x7f6a900a98f0 legacy=0x7f6a90138b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:05.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.932+0000 7f6a777fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:05.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.932+0000 7f6a777fe640 1 monclient: found mon.b 2024-03-25T18:18:05.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.932+0000 7f6a777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:05.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.932+0000 7f6a777fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:18:05.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.932+0000 7f6a777fe640 1 -- 172.21.15.63:0/2811392052 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a900b09e0 con 0x7f6a9013f710 2024-03-25T18:18:05.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.932+0000 7f6a777fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:05.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.932+0000 7f6a777fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:05.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.932+0000 7f6a777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:05.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.932+0000 7f6a954d3640 5 monclient: authenticate success, global_id 4480 2024-03-25T18:18:05.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.932+0000 7f6a954d3640 10 monclient: _renew_subs 2024-03-25T18:18:05.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.932+0000 7f6a954d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:18:05.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.932+0000 7f6a954d3640 1 -- 172.21.15.63:0/2811392052 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a900b0c30 con 0x7f6a9013f710 2024-03-25T18:18:05.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.933+0000 7f6a954d3640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:05.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.933+0000 7f6a954d3640 10 monclient: _renew_subs 2024-03-25T18:18:05.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.933+0000 7f6a954d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:18:05.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.933+0000 7f6a954d3640 1 -- 172.21.15.63:0/2811392052 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a900acb20 con 0x7f6a9013f710 2024-03-25T18:18:05.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.933+0000 7f6a954d3640 1 librados: init done 2024-03-25T18:18:05.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.934+0000 7f6a954d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:05.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.934+0000 7f6a954d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:05.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.934+0000 7f6a954d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:18:05.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.934+0000 7f6a954d3640 1 -- 172.21.15.63:0/2811392052 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a900ace20 con 0x7f6a9013f710 2024-03-25T18:18:05.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.953+0000 7f6a777fe640 1 -- 172.21.15.63:0/2811392052 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a84001ce0 con 0x7f6a9013f710 2024-03-25T18:18:05.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.953+0000 7f6a777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:05.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.953+0000 7f6a777fe640 1 -- 172.21.15.63:0/2811392052 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f6a840038a0 con 0x7f6a9013f710 2024-03-25T18:18:05.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.953+0000 7f6a777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:05.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.953+0000 7f6a777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:05.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.953+0000 7f6a777fe640 10 monclient: dump: 2024-03-25T18:18:05.954 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:05.955 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:05.955 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:05.955 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:05.955 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:05.955 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:05.955 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:05.955 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:05.955 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:05.955 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:05.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.953+0000 7f6a8d7fa640 4 set_mon_vals no callback set 2024-03-25T18:18:05.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.954+0000 7f6a777fe640 1 -- 172.21.15.63:0/2811392052 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f6a84003ac0 con 0x7f6a9013f710 2024-03-25T18:18:05.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.954+0000 7f6a8effd640 10 monclient: get_auth_request con 0x7f6a7c06dc20 auth_method 0 2024-03-25T18:18:05.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.955+0000 7f6a777fe640 10 client.4480.objecter ms_handle_connect 0x7f6a7c06dc20 2024-03-25T18:18:05.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.956+0000 7f6a777fe640 1 -- 172.21.15.63:0/2811392052 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f6a840547d0 con 0x7f6a9013f710 2024-03-25T18:18:05.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.956+0000 7f6a777fe640 10 client.4480.objecter ms_dispatch 0x7f6a90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:05.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.956+0000 7f6a777fe640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:05.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.956+0000 7f6a777fe640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:05.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:05.956+0000 7f6a777fe640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-25T18:18:06.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.208+0000 7f6a777fe640 1 -- 172.21.15.63:0/2811392052 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a84050110 con 0x7f6a9013f710 2024-03-25T18:18:06.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.208+0000 7f6a777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:06.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.208+0000 7f6a777fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:06.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.469+0000 7f6a954d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:06.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.470+0000 7f6a954d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:06.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.470+0000 7f6a954d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:18:06.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.470+0000 7f6a954d3640 1 -- 172.21.15.63:0/2811392052 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a901395c0 con 0x7f6a9013f710 2024-03-25T18:18:06.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.488+0000 7f6a777fe640 1 -- 172.21.15.63:0/2811392052 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f6a901395c0 con 0x7f6a9013f710 2024-03-25T18:18:06.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.488+0000 7f6a777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:06.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.488+0000 7f6a777fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:06.490 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:18:06.490 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:06.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.490+0000 7f6a954d3640 10 librados: watch_flush enter 2024-03-25T18:18:06.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.490+0000 7f6a954d3640 10 librados: watch_flush exit 2024-03-25T18:18:06.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.491+0000 7f6a954d3640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-25T18:18:06.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.491+0000 7f6a954d3640 10 client.4480.objecter successfully canceled tick 2024-03-25T18:18:06.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.491+0000 7f6a954d3640 1 -- 172.21.15.63:0/2811392052 >> v1:172.21.15.81:6800/3640340873 conn(0x7f6a7c06dc20 legacy=0x7f6a7c0700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:06.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.491+0000 7f6a954d3640 10 monclient: shutdown 2024-03-25T18:18:06.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.491+0000 7f6a954d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:06.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.491+0000 7f6a954d3640 1 -- 172.21.15.63:0/2811392052 >> v1:172.21.15.81:6789/0 conn(0x7f6a9013f710 legacy=0x7f6a90135880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:06.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.491+0000 7f6a954d3640 1 -- 172.21.15.63:0/2811392052 shutdown_connections 2024-03-25T18:18:06.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.491+0000 7f6a954d3640 1 -- 172.21.15.63:0/2811392052 >> 172.21.15.63:0/2811392052 conn(0x7f6a900b7ea0 msgr2=0x7f6a90074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:06.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.491+0000 7f6a954d3640 1 -- 172.21.15.63:0/2811392052 shutdown_connections 2024-03-25T18:18:06.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.491+0000 7f6a954d3640 1 -- 172.21.15.63:0/2811392052 wait complete. 2024-03-25T18:18:06.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:06.491+0000 7f6a954d3640 1 librados: shutdown 2024-03-25T18:18:06.503 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:13.310 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:13.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.415+0000 7fb4b81e6640 10 monclient: get_monmap_and_config 2024-03-25T18:18:13.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.415+0000 7fb4b81e6640 10 monclient: build_initial_monmap 2024-03-25T18:18:13.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.415+0000 7fb4b81e6640 10 monclient: monmap: 2024-03-25T18:18:13.416 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:18:13.416 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:13.416 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:18:13.416391+0000 2024-03-25T18:18:13.417 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:18:13.416391+0000 2024-03-25T18:18:13.417 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:13.417 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:13.417 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:18:13.417 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:18:13.417 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:18:13.417 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:13.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.415+0000 7fb4b81e6640 1 Processor -- start 2024-03-25T18:18:13.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.416+0000 7fb4b81e6640 1 -- start start 2024-03-25T18:18:13.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.416+0000 7fb4b81e6640 10 monclient: init 2024-03-25T18:18:13.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.416+0000 7fb4b81e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:13.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.416+0000 7fb4b81e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:13.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.416+0000 7fb4b81e6640 10 monclient(hunting): picked mon.noname-a con 0x7fb4b00b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T18:18:13.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.416+0000 7fb4b81e6640 10 monclient(hunting): picked mon.noname-c con 0x7fb4b00b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:18:13.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.416+0000 7fb4b81e6640 10 monclient(hunting): picked mon.noname-b con 0x7fb4b014be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:18:13.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.416+0000 7fb4b81e6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b00af940 con 0x7fb4b00b4d60 2024-03-25T18:18:13.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.416+0000 7fb4b81e6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b00b0a70 con 0x7fb4b00b1730 2024-03-25T18:18:13.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.416+0000 7fb4b81e6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b00b0d50 con 0x7fb4b014be90 2024-03-25T18:18:13.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.416+0000 7fb4b81e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:13.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.416+0000 7fb4b81e6640 10 monclient(hunting): authenticate will time out at 7787.602985s 2024-03-25T18:18:13.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.417+0000 7fb4b6783640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fb4b014be90 0x7fb4b014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:34200/0 (socket says 172.21.15.63:34200) 2024-03-25T18:18:13.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.417+0000 7fb4b6783640 1 -- 172.21.15.63:0/4159153393 learned_addr learned my addr 172.21.15.63:0/4159153393 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:18:13.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.449+0000 7fb4b4f80640 1 -- 172.21.15.63:0/4159153393 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341585474 0 0) 0x7fb4b00af940 con 0x7fb4b00b4d60 2024-03-25T18:18:13.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.449+0000 7fb4b4f80640 10 monclient(hunting): _negotiate 2024-03-25T18:18:13.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.449+0000 7fb4b4f80640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:13.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.449+0000 7fb4b4f80640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:13.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.449+0000 7fb4b4f80640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:18:13.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.449+0000 7fb4b4f80640 1 -- 172.21.15.63:0/4159153393 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb49c003520 con 0x7fb4b00b4d60 2024-03-25T18:18:13.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b4f80640 1 -- 172.21.15.63:0/4159153393 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3956033761 0 0) 0x7fb49c003520 con 0x7fb4b00b4d60 2024-03-25T18:18:13.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b4f80640 1 -- 172.21.15.63:0/4159153393 >> v1:172.21.15.63:6790/0 conn(0x7fb4b014be90 legacy=0x7fb4b014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:13.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b4f80640 1 -- 172.21.15.63:0/4159153393 >> v1:172.21.15.81:6789/0 conn(0x7fb4b00b1730 legacy=0x7fb4b00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:13.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b4f80640 10 monclient: _finish_hunting 0 2024-03-25T18:18:13.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b4f80640 1 monclient: found mon.noname-a 2024-03-25T18:18:13.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b4f80640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:13.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b4f80640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:18:13.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b4f80640 1 -- 172.21.15.63:0/4159153393 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b00b1030 con 0x7fb4b00b4d60 2024-03-25T18:18:13.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b4f80640 10 monclient: _finish_auth 0 2024-03-25T18:18:13.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b4f80640 10 monclient: _check_auth_tickets 2024-03-25T18:18:13.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b4f80640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:13.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b4f80640 1 -- 172.21.15.63:0/4159153393 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fb4ac001690 con 0x7fb4b00b4d60 2024-03-25T18:18:13.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b4f80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:13.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b4f80640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:13.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b4f80640 10 monclient: dump: 2024-03-25T18:18:13.468 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:13.468 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:13.468 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:13.468 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:13.468 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:13.468 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:13.468 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:13.468 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:13.468 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:13.469 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:13.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b81e6640 5 monclient: authenticate success, global_id 4458 2024-03-25T18:18:13.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.465+0000 7fb4b81e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:13.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.480+0000 7fb4b4f80640 1 -- 172.21.15.63:0/4159153393 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4ac0017f0 con 0x7fb4b00b4d60 2024-03-25T18:18:13.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.480+0000 7fb4b4f80640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:13.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.480+0000 7fb4b4f80640 1 -- 172.21.15.63:0/4159153393 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fb4ac001c00 con 0x7fb4b00b4d60 2024-03-25T18:18:13.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.480+0000 7fb4b4f80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:13.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.480+0000 7fb4b4f80640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:13.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.480+0000 7fb4b4f80640 10 monclient: dump: 2024-03-25T18:18:13.482 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:13.482 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:13.482 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:13.482 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:13.482 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:13.482 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:13.482 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:13.482 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:13.482 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:13.483 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:13.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.480+0000 7fb4b81e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:13.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.480+0000 7fb4b81e6640 4 set_mon_vals no callback set 2024-03-25T18:18:13.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.481+0000 7fb4b81e6640 10 monclient: shutdown 2024-03-25T18:18:13.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.481+0000 7fb4b81e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:13.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.481+0000 7fb4b81e6640 1 -- 172.21.15.63:0/4159153393 >> v1:172.21.15.63:6789/0 conn(0x7fb4b00b4d60 legacy=0x7fb4b013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:13.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.481+0000 7fb4b81e6640 1 -- 172.21.15.63:0/4159153393 shutdown_connections 2024-03-25T18:18:13.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.481+0000 7fb4b81e6640 1 -- 172.21.15.63:0/4159153393 >> 172.21.15.63:0/4159153393 conn(0x7fb4b00617a0 msgr2=0x7fb4b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:13.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.481+0000 7fb4b81e6640 1 -- 172.21.15.63:0/4159153393 shutdown_connections 2024-03-25T18:18:13.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.481+0000 7fb4b81e6640 1 -- 172.21.15.63:0/4159153393 wait complete. 2024-03-25T18:18:13.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.482+0000 7fb4b81e6640 10 monclient: build_initial_monmap 2024-03-25T18:18:13.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.482+0000 7fb4b81e6640 10 monclient: monmap: 2024-03-25T18:18:13.485 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:18:13.485 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:13.485 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:18:13.485 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:18:13.486 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:13.486 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:18:13.486 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:18:13.486 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:18:13.486 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:18:13.486 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:13.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.483+0000 7fb4b81e6640 1 librados: starting msgr at 2024-03-25T18:18:13.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.483+0000 7fb4b81e6640 1 librados: starting objecter 2024-03-25T18:18:13.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.483+0000 7fb4b81e6640 1 Processor -- start 2024-03-25T18:18:13.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.483+0000 7fb4b81e6640 1 -- start start 2024-03-25T18:18:13.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.483+0000 7fb4b81e6640 1 librados: setting wanted keys 2024-03-25T18:18:13.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.483+0000 7fb4b81e6640 1 librados: calling monclient init 2024-03-25T18:18:13.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.483+0000 7fb4b81e6640 10 monclient: init 2024-03-25T18:18:13.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.484+0000 7fb4b81e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:13.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.484+0000 7fb4b81e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:13.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.484+0000 7fb4b81e6640 10 monclient(hunting): picked mon.noname-b con 0x7fb4b00b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:18:13.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.484+0000 7fb4b81e6640 10 monclient(hunting): picked mon.noname-a con 0x7fb4b00b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T18:18:13.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.484+0000 7fb4b81e6640 10 monclient(hunting): picked mon.noname-c con 0x7fb4b014be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:18:13.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.484+0000 7fb4b81e6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b0142f70 con 0x7fb4b00b4d60 2024-03-25T18:18:13.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.484+0000 7fb4b81e6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b01411b0 con 0x7fb4b00b1730 2024-03-25T18:18:13.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.484+0000 7fb4b81e6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b0143250 con 0x7fb4b014be90 2024-03-25T18:18:13.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.484+0000 7fb4b81e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:13.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.484+0000 7fb4b81e6640 10 monclient(hunting): authenticate will time out at 7787.670649s 2024-03-25T18:18:13.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.494+0000 7fb4b5781640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fb4b00b4d60 0x7fb4b013d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:42366/0 (socket says 172.21.15.63:42366) 2024-03-25T18:18:13.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.494+0000 7fb4b5781640 1 -- 172.21.15.63:0/664677738 learned_addr learned my addr 172.21.15.63:0/664677738 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:18:13.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.495+0000 7fb4a67fc640 10 client.?.objecter ms_handle_connect 0x7fb4b00b4d60 2024-03-25T18:18:13.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.495+0000 7fb4a67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:13.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.508+0000 7fb4a67fc640 10 client.?.objecter ms_handle_connect 0x7fb4b014be90 2024-03-25T18:18:13.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.508+0000 7fb4a67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:13.519 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.518+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429811389 0 0) 0x7fb4b0142f70 con 0x7fb4b00b4d60 2024-03-25T18:18:13.519 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.518+0000 7fb4a67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:13.519 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.518+0000 7fb4a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:13.519 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.518+0000 7fb4a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:13.519 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.518+0000 7fb4a67fc640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:18:13.520 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.518+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb494004eb0 con 0x7fb4b00b4d60 2024-03-25T18:18:13.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.526+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685530273 0 0) 0x7fb4b0143250 con 0x7fb4b014be90 2024-03-25T18:18:13.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.526+0000 7fb4a67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:13.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.526+0000 7fb4a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:13.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.526+0000 7fb4a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:13.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.526+0000 7fb4a67fc640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:18:13.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.526+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4b0142f70 con 0x7fb4b014be90 2024-03-25T18:18:13.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.527+0000 7fb4a67fc640 10 client.?.objecter ms_handle_connect 0x7fb4b00b1730 2024-03-25T18:18:13.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.527+0000 7fb4a67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:13.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.530+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3158320312 0 0) 0x7fb494004eb0 con 0x7fb4b00b4d60 2024-03-25T18:18:13.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.530+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4b0143250 con 0x7fb4b00b4d60 2024-03-25T18:18:13.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.530+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fb4a00019d0 con 0x7fb4b00b4d60 2024-03-25T18:18:13.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.530+0000 7fb4a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:13.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.530+0000 7fb4a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:13.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.530+0000 7fb4a67fc640 10 monclient(hunting): dump: 2024-03-25T18:18:13.532 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:13.532 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:13.532 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:13.532 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:13.532 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:13.532 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:13.532 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:13.532 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:13.532 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:13.532 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:13.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.530+0000 7fb4a67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:13.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.539+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3247750153 0 0) 0x7fb4b0142f70 con 0x7fb4b014be90 2024-03-25T18:18:13.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.539+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb494004eb0 con 0x7fb4b014be90 2024-03-25T18:18:13.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.539+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fb4a8001b00 con 0x7fb4b014be90 2024-03-25T18:18:13.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.539+0000 7fb4a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:13.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.539+0000 7fb4a67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:13.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.539+0000 7fb4a67fc640 10 monclient(hunting): dump: 2024-03-25T18:18:13.541 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:13.541 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:13.541 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:13.541 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:13.541 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:13.541 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:13.541 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:13.541 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:13.541 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:13.541 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:13.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.543+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361908638 0 0) 0x7fb4b01411b0 con 0x7fb4b00b1730 2024-03-25T18:18:13.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.543+0000 7fb4a67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:13.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.543+0000 7fb4a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:13.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.543+0000 7fb4a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:13.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.543+0000 7fb4a67fc640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:18:13.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.543+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4b0142f70 con 0x7fb4b00b1730 2024-03-25T18:18:13.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1386800167 0 0) 0x7fb4b0143250 con 0x7fb4b00b4d60 2024-03-25T18:18:13.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 >> v1:172.21.15.63:6790/0 conn(0x7fb4b014be90 legacy=0x7fb4b0141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:13.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 >> v1:172.21.15.81:6789/0 conn(0x7fb4b00b1730 legacy=0x7fb4b01372b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:13.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4a67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:13.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4a67fc640 1 monclient: found mon.a 2024-03-25T18:18:13.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:13.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4a67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:18:13.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b0140c90 con 0x7fb4b00b4d60 2024-03-25T18:18:13.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4a67fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:13.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4a67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:13.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:13.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4b81e6640 5 monclient: authenticate success, global_id 4461 2024-03-25T18:18:13.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4b81e6640 10 monclient: _renew_subs 2024-03-25T18:18:13.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4b81e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:18:13.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4b81e6640 1 -- 172.21.15.63:0/664677738 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b01379c0 con 0x7fb4b00b4d60 2024-03-25T18:18:13.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4b81e6640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:13.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4b81e6640 10 monclient: _renew_subs 2024-03-25T18:18:13.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4b81e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:18:13.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4b81e6640 1 -- 172.21.15.63:0/664677738 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b013d9e0 con 0x7fb4b00b4d60 2024-03-25T18:18:13.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.551+0000 7fb4b81e6640 1 librados: init done 2024-03-25T18:18:13.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.552+0000 7fb4b81e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:13.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.552+0000 7fb4b81e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:13.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.552+0000 7fb4b81e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:18:13.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.552+0000 7fb4b81e6640 1 -- 172.21.15.63:0/664677738 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b00627b0 con 0x7fb4b00b4d60 2024-03-25T18:18:13.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.870+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4a0003610 con 0x7fb4b00b4d60 2024-03-25T18:18:13.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.870+0000 7fb4a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:13.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.870+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fb4a00039a0 con 0x7fb4b00b4d60 2024-03-25T18:18:13.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.870+0000 7fb4a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:13.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.870+0000 7fb4a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:13.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.870+0000 7fb4a67fc640 10 monclient: dump: 2024-03-25T18:18:13.872 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:13.872 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:13.872 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:13.872 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:13.872 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:13.873 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:13.873 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:13.873 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:13.873 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:13.873 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:13.873 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.870+0000 7fb4a7fff640 4 set_mon_vals no callback set 2024-03-25T18:18:13.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.871+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fb4a0003bc0 con 0x7fb4b00b4d60 2024-03-25T18:18:13.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.872+0000 7fb4b5f82640 10 monclient: get_auth_request con 0x7fb49406df40 auth_method 0 2024-03-25T18:18:13.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.872+0000 7fb4a67fc640 10 client.4461.objecter ms_handle_connect 0x7fb49406df40 2024-03-25T18:18:13.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.879+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fb4a00547e0 con 0x7fb4b00b4d60 2024-03-25T18:18:13.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.879+0000 7fb4a67fc640 10 client.4461.objecter ms_dispatch 0x7fb4b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:13.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.879+0000 7fb4a67fc640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:13.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.879+0000 7fb4a67fc640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:13.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.879+0000 7fb4a67fc640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-25T18:18:13.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.879+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb4a00502f0 con 0x7fb4b00b4d60 2024-03-25T18:18:13.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.879+0000 7fb4a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:13.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:13.879+0000 7fb4a67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:14.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.137+0000 7fb4b81e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:14.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.137+0000 7fb4b81e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:14.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.137+0000 7fb4b81e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:18:14.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.137+0000 7fb4b81e6640 1 -- 172.21.15.63:0/664677738 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4b014bbf0 con 0x7fb4b00b4d60 2024-03-25T18:18:14.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.166+0000 7fb4a67fc640 1 -- 172.21.15.63:0/664677738 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fb4a0052220 con 0x7fb4b00b4d60 2024-03-25T18:18:14.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.166+0000 7fb4a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:14.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.166+0000 7fb4a67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:14.167 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:18:14.167 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:14.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.168+0000 7fb4b81e6640 10 librados: watch_flush enter 2024-03-25T18:18:14.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.168+0000 7fb4b81e6640 10 librados: watch_flush exit 2024-03-25T18:18:14.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.168+0000 7fb4b81e6640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-25T18:18:14.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.168+0000 7fb4b81e6640 10 client.4461.objecter successfully canceled tick 2024-03-25T18:18:14.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.168+0000 7fb4b81e6640 1 -- 172.21.15.63:0/664677738 >> v1:172.21.15.81:6800/3640340873 conn(0x7fb49406df40 legacy=0x7fb494070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:14.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.168+0000 7fb4b81e6640 10 monclient: shutdown 2024-03-25T18:18:14.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.168+0000 7fb4b81e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:14.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.168+0000 7fb4b81e6640 1 -- 172.21.15.63:0/664677738 >> v1:172.21.15.63:6789/0 conn(0x7fb4b00b4d60 legacy=0x7fb4b013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:14.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.168+0000 7fb4b81e6640 1 -- 172.21.15.63:0/664677738 shutdown_connections 2024-03-25T18:18:14.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.168+0000 7fb4b81e6640 1 -- 172.21.15.63:0/664677738 >> 172.21.15.63:0/664677738 conn(0x7fb4b00617a0 msgr2=0x7fb4b00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:14.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.168+0000 7fb4b81e6640 1 -- 172.21.15.63:0/664677738 shutdown_connections 2024-03-25T18:18:14.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.168+0000 7fb4b81e6640 1 -- 172.21.15.63:0/664677738 wait complete. 2024-03-25T18:18:14.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:14.168+0000 7fb4b81e6640 1 librados: shutdown 2024-03-25T18:18:14.180 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:19.046 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:00.738 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:19.106 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:00.798 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:19.144 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:00.836 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:19.147 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:06:00.808 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:19.158 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:00.850 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:19.162 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:06:00.823 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:19.170 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:00.862 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:19.184 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:06:00.845 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:19.206 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:00.898 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:20.987 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:21.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.100+0000 7f252f686640 10 monclient: get_monmap_and_config 2024-03-25T18:18:21.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.100+0000 7f252f686640 10 monclient: build_initial_monmap 2024-03-25T18:18:21.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.100+0000 7f252f686640 10 monclient: monmap: 2024-03-25T18:18:21.100 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:18:21.100 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:21.100 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:18:21.100561+0000 2024-03-25T18:18:21.100 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:18:21.100561+0000 2024-03-25T18:18:21.101 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:21.101 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:21.101 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:18:21.101 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:18:21.101 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:18:21.101 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:21.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.101+0000 7f252f686640 1 Processor -- start 2024-03-25T18:18:21.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.101+0000 7f252f686640 1 -- start start 2024-03-25T18:18:21.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.101+0000 7f252f686640 10 monclient: init 2024-03-25T18:18:21.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.102+0000 7f252f686640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:21.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.102+0000 7f252f686640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:21.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.102+0000 7f252f686640 10 monclient(hunting): picked mon.noname-a con 0x7f25280a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:18:21.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.102+0000 7f252f686640 10 monclient(hunting): picked mon.noname-c con 0x7f252813f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:18:21.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.102+0000 7f252f686640 10 monclient(hunting): picked mon.noname-b con 0x7f25281430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:18:21.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.102+0000 7f252f686640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2528142520 con 0x7f25280a98f0 2024-03-25T18:18:21.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.102+0000 7f252f686640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25280b28a0 con 0x7f252813f710 2024-03-25T18:18:21.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.102+0000 7f252f686640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25280b2a30 con 0x7f25281430c0 2024-03-25T18:18:21.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.102+0000 7f252f686640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:21.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.102+0000 7f252f686640 10 monclient(hunting): authenticate will time out at 7795.287547s 2024-03-25T18:18:21.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.110+0000 7f252dc23640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f25281430c0 0x7f252814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:49358/0 (socket says 172.21.15.63:49358) 2024-03-25T18:18:21.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.110+0000 7f252dc23640 1 -- 172.21.15.63:0/2544690295 learned_addr learned my addr 172.21.15.63:0/2544690295 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:18:21.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.123+0000 7f251ffff640 1 -- 172.21.15.63:0/2544690295 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689206500 0 0) 0x7f25280b28a0 con 0x7f252813f710 2024-03-25T18:18:21.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.123+0000 7f251ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:21.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.123+0000 7f251ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:21.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.123+0000 7f251ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:21.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.123+0000 7f251ffff640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:18:21.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.123+0000 7f251ffff640 1 -- 172.21.15.63:0/2544690295 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2514003520 con 0x7f252813f710 2024-03-25T18:18:21.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.126+0000 7f251ffff640 1 -- 172.21.15.63:0/2544690295 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155157348 0 0) 0x7f25280b2a30 con 0x7f25281430c0 2024-03-25T18:18:21.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.126+0000 7f251ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:21.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.126+0000 7f251ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:21.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.126+0000 7f251ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:21.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.126+0000 7f251ffff640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:18:21.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.126+0000 7f251ffff640 1 -- 172.21.15.63:0/2544690295 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25280b28a0 con 0x7f25281430c0 2024-03-25T18:18:21.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f251ffff640 1 -- 172.21.15.63:0/2544690295 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 141006600 0 0) 0x7f2514003520 con 0x7f252813f710 2024-03-25T18:18:21.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f251ffff640 1 -- 172.21.15.63:0/2544690295 >> v1:172.21.15.63:6790/0 conn(0x7f25281430c0 legacy=0x7f252814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:21.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f251ffff640 1 -- 172.21.15.63:0/2544690295 >> v1:172.21.15.63:6789/0 conn(0x7f25280a98f0 legacy=0x7f25280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:18:21.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f251ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:21.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f251ffff640 1 monclient: found mon.noname-c 2024-03-25T18:18:21.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f251ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:21.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f251ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:18:21.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f251ffff640 1 -- 172.21.15.63:0/2544690295 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25280b44c0 con 0x7f252813f710 2024-03-25T18:18:21.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f251ffff640 10 monclient: _finish_auth 0 2024-03-25T18:18:21.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f251ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:21.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f251ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:21.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f251ffff640 1 -- 172.21.15.63:0/2544690295 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2518001690 con 0x7f252813f710 2024-03-25T18:18:21.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f252f686640 5 monclient: authenticate success, global_id 4489 2024-03-25T18:18:21.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f252f686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:21.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f251ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:21.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f251ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:21.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f251ffff640 10 monclient: dump: 2024-03-25T18:18:21.136 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:21.136 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:21.136 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:21.136 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:21.136 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:21.136 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:21.136 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:21.136 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:21.136 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:21.136 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:21.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.134+0000 7f252f686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:21.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.138+0000 7f251ffff640 1 -- 172.21.15.63:0/2544690295 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25180017f0 con 0x7f252813f710 2024-03-25T18:18:21.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.138+0000 7f251ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:21.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.138+0000 7f251ffff640 1 -- 172.21.15.63:0/2544690295 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2518001c00 con 0x7f252813f710 2024-03-25T18:18:21.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.138+0000 7f252f686640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:21.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.138+0000 7f252f686640 4 set_mon_vals no callback set 2024-03-25T18:18:21.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.138+0000 7f251ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:21.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.138+0000 7f251ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:21.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.138+0000 7f251ffff640 10 monclient: dump: 2024-03-25T18:18:21.139 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:21.139 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:21.139 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:21.139 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:21.139 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:21.139 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:21.139 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:21.139 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:21.139 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:21.140 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:21.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.139+0000 7f252f686640 10 monclient: shutdown 2024-03-25T18:18:21.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.139+0000 7f252f686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:21.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.139+0000 7f252f686640 1 -- 172.21.15.63:0/2544690295 >> v1:172.21.15.81:6789/0 conn(0x7f252813f710 legacy=0x7f25281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:21.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.139+0000 7f252f686640 1 -- 172.21.15.63:0/2544690295 shutdown_connections 2024-03-25T18:18:21.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.139+0000 7f252f686640 1 -- 172.21.15.63:0/2544690295 >> 172.21.15.63:0/2544690295 conn(0x7f25280b7ea0 msgr2=0x7f25280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:21.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.139+0000 7f252f686640 1 -- 172.21.15.63:0/2544690295 shutdown_connections 2024-03-25T18:18:21.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.139+0000 7f252f686640 1 -- 172.21.15.63:0/2544690295 wait complete. 2024-03-25T18:18:21.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.140+0000 7f252f686640 10 monclient: build_initial_monmap 2024-03-25T18:18:21.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.140+0000 7f252f686640 10 monclient: monmap: 2024-03-25T18:18:21.142 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:18:21.142 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:21.142 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:18:21.142 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:18:21.143 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:21.143 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:18:21.143 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:18:21.143 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:18:21.143 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:18:21.143 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:21.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.140+0000 7f252f686640 1 librados: starting msgr at 2024-03-25T18:18:21.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.140+0000 7f252f686640 1 librados: starting objecter 2024-03-25T18:18:21.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.140+0000 7f252f686640 1 Processor -- start 2024-03-25T18:18:21.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.140+0000 7f252f686640 1 -- start start 2024-03-25T18:18:21.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.140+0000 7f252f686640 1 librados: setting wanted keys 2024-03-25T18:18:21.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.140+0000 7f252f686640 1 librados: calling monclient init 2024-03-25T18:18:21.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.140+0000 7f252f686640 10 monclient: init 2024-03-25T18:18:21.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.141+0000 7f252f686640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:21.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.141+0000 7f252f686640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:21.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.141+0000 7f252f686640 10 monclient(hunting): picked mon.noname-c con 0x7f25280a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:18:21.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.141+0000 7f252f686640 10 monclient(hunting): picked mon.noname-b con 0x7f252813f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:18:21.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.141+0000 7f252f686640 10 monclient(hunting): picked mon.noname-a con 0x7f25281430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:18:21.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.141+0000 7f252f686640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25281db660 con 0x7f25281430c0 2024-03-25T18:18:21.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.141+0000 7f252f686640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25281db7f0 con 0x7f252813f710 2024-03-25T18:18:21.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.141+0000 7f252f686640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252814dee0 con 0x7f25280a98f0 2024-03-25T18:18:21.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.141+0000 7f252f686640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:21.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.141+0000 7f252f686640 10 monclient(hunting): authenticate will time out at 7795.326807s 2024-03-25T18:18:21.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.147+0000 7f252cc21640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f252813f710 0x7f25280ae760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:34752/0 (socket says 172.21.15.63:34752) 2024-03-25T18:18:21.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.147+0000 7f252cc21640 1 -- 172.21.15.63:0/3664503072 learned_addr learned my addr 172.21.15.63:0/3664503072 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:18:21.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.148+0000 7f251dffb640 10 client.?.objecter ms_handle_connect 0x7f252813f710 2024-03-25T18:18:21.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.148+0000 7f251dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:21.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.160+0000 7f251dffb640 10 client.?.objecter ms_handle_connect 0x7f25281430c0 2024-03-25T18:18:21.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.160+0000 7f251dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:21.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.162+0000 7f251dffb640 10 client.?.objecter ms_handle_connect 0x7f25280a98f0 2024-03-25T18:18:21.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.162+0000 7f251dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:21.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.167+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770003106 0 0) 0x7f25281db660 con 0x7f25281430c0 2024-03-25T18:18:21.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.167+0000 7f251dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:21.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.167+0000 7f251dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:21.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.167+0000 7f251dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:21.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.167+0000 7f251dffb640 10 monclient(hunting): my global_id is 4464 2024-03-25T18:18:21.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.167+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f250c004eb0 con 0x7f25281430c0 2024-03-25T18:18:21.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.167+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868486814 0 0) 0x7f25281db7f0 con 0x7f252813f710 2024-03-25T18:18:21.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.167+0000 7f251dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:21.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.167+0000 7f251dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:21.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.167+0000 7f251dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:21.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.167+0000 7f251dffb640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:18:21.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.167+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25281db660 con 0x7f252813f710 2024-03-25T18:18:21.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.173+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294341916 0 0) 0x7f252814dee0 con 0x7f25280a98f0 2024-03-25T18:18:21.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.173+0000 7f251dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:21.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.173+0000 7f251dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:21.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.173+0000 7f251dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:21.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.173+0000 7f251dffb640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:18:21.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.173+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25281db7f0 con 0x7f25280a98f0 2024-03-25T18:18:21.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.176+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1710617278 0 0) 0x7f250c004eb0 con 0x7f25281430c0 2024-03-25T18:18:21.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.176+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f252814dee0 con 0x7f25281430c0 2024-03-25T18:18:21.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.176+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2520001bd0 con 0x7f25281430c0 2024-03-25T18:18:21.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.176+0000 7f251dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:21.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.176+0000 7f251dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:21.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.176+0000 7f251dffb640 10 monclient(hunting): dump: 2024-03-25T18:18:21.177 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:21.177 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:21.177 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:21.177 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:21.177 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:21.177 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:21.177 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:21.177 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:21.177 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:21.177 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:21.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.176+0000 7f251dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:21.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.181+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1332094445 0 0) 0x7f25281db660 con 0x7f252813f710 2024-03-25T18:18:21.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.181+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f250c004eb0 con 0x7f252813f710 2024-03-25T18:18:21.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.181+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2518001690 con 0x7f252813f710 2024-03-25T18:18:21.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.181+0000 7f251dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:21.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.181+0000 7f251dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:21.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.181+0000 7f251dffb640 10 monclient(hunting): dump: 2024-03-25T18:18:21.182 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:21.182 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:21.182 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:21.182 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:21.182 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:21.182 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:21.182 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:21.182 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:21.183 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:21.183 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:21.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3070763407 0 0) 0x7f252814dee0 con 0x7f25281430c0 2024-03-25T18:18:21.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 >> v1:172.21.15.63:6790/0 conn(0x7f25280a98f0 legacy=0x7f25281dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:21.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 >> v1:172.21.15.81:6789/0 conn(0x7f252813f710 legacy=0x7f25280ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:21.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f251dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:21.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f251dffb640 1 monclient: found mon.a 2024-03-25T18:18:21.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f251dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:21.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f251dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:18:21.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252814e070 con 0x7f25281430c0 2024-03-25T18:18:21.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f251dffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:21.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f251dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:21.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f251dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:21.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f252f686640 5 monclient: authenticate success, global_id 4464 2024-03-25T18:18:21.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f252f686640 10 monclient: _renew_subs 2024-03-25T18:18:21.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f252f686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:18:21.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f252f686640 1 -- 172.21.15.63:0/3664503072 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25280b11e0 con 0x7f25281430c0 2024-03-25T18:18:21.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f252f686640 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:21.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f252f686640 10 monclient: _renew_subs 2024-03-25T18:18:21.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f252f686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:18:21.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f252f686640 1 -- 172.21.15.63:0/3664503072 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25281e3470 con 0x7f25281430c0 2024-03-25T18:18:21.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.187+0000 7f252f686640 1 librados: init done 2024-03-25T18:18:21.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.188+0000 7f252f686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:21.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.188+0000 7f252f686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:21.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.188+0000 7f252f686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:18:21.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.188+0000 7f252f686640 1 -- 172.21.15.63:0/3664503072 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f252814e260 con 0x7f25281430c0 2024-03-25T18:18:21.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.515+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2520002080 con 0x7f25281430c0 2024-03-25T18:18:21.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.515+0000 7f251dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:21.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.515+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f2520004d00 con 0x7f25281430c0 2024-03-25T18:18:21.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.515+0000 7f251dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:21.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.515+0000 7f251dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:21.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.515+0000 7f251dffb640 10 monclient: dump: 2024-03-25T18:18:21.515 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:21.515 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:21.515 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:21.515 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:21.515 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:21.516 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:21.516 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:21.516 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:21.516 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:21.516 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:21.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.515+0000 7f251f7fe640 4 set_mon_vals no callback set 2024-03-25T18:18:21.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.515+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f252001acd0 con 0x7f25281430c0 2024-03-25T18:18:21.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.516+0000 7f252d422640 10 monclient: get_auth_request con 0x7f250c06df20 auth_method 0 2024-03-25T18:18:21.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.516+0000 7f251dffb640 10 client.4464.objecter ms_handle_connect 0x7f250c06df20 2024-03-25T18:18:21.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.525+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f2520054a10 con 0x7f25281430c0 2024-03-25T18:18:21.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.525+0000 7f251dffb640 10 client.4464.objecter ms_dispatch 0x7f2528000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:21.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.525+0000 7f251dffb640 3 client.4464.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:21.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.525+0000 7f251dffb640 3 client.4464.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:21.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.525+0000 7f251dffb640 20 client.4464.objecter dump_active .. 0 homeless 2024-03-25T18:18:21.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.525+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2520050350 con 0x7f25281430c0 2024-03-25T18:18:21.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.525+0000 7f251dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:21.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.525+0000 7f251dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:21.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.746+0000 7f252f686640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:21.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.746+0000 7f252f686640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:21.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.746+0000 7f252f686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:18:21.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.746+0000 7f252f686640 1 -- 172.21.15.63:0/3664503072 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25280b1760 con 0x7f25281430c0 2024-03-25T18:18:21.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.769+0000 7f251dffb640 1 -- 172.21.15.63:0/3664503072 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f2520052450 con 0x7f25281430c0 2024-03-25T18:18:21.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.769+0000 7f251dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:21.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.769+0000 7f251dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:21.769 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:18:21.769 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:21.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.771+0000 7f252f686640 10 librados: watch_flush enter 2024-03-25T18:18:21.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.771+0000 7f252f686640 10 librados: watch_flush exit 2024-03-25T18:18:21.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.771+0000 7f252f686640 20 client.4464.objecter shutdown clearing up homeless session... 2024-03-25T18:18:21.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.771+0000 7f252f686640 10 client.4464.objecter successfully canceled tick 2024-03-25T18:18:21.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.771+0000 7f252f686640 1 -- 172.21.15.63:0/3664503072 >> v1:172.21.15.81:6800/3640340873 conn(0x7f250c06df20 legacy=0x7f250c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:21.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.771+0000 7f252f686640 10 monclient: shutdown 2024-03-25T18:18:21.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.771+0000 7f252f686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:21.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.771+0000 7f252f686640 1 -- 172.21.15.63:0/3664503072 >> v1:172.21.15.63:6789/0 conn(0x7f25281430c0 legacy=0x7f25281e1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:21.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.771+0000 7f252f686640 1 -- 172.21.15.63:0/3664503072 shutdown_connections 2024-03-25T18:18:21.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.771+0000 7f252f686640 1 -- 172.21.15.63:0/3664503072 >> 172.21.15.63:0/3664503072 conn(0x7f25280b7ea0 msgr2=0x7f2528074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:21.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.771+0000 7f252f686640 1 -- 172.21.15.63:0/3664503072 shutdown_connections 2024-03-25T18:18:21.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.771+0000 7f252f686640 1 -- 172.21.15.63:0/3664503072 wait complete. 2024-03-25T18:18:21.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:21.771+0000 7f252f686640 1 librados: shutdown 2024-03-25T18:18:21.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:28.591 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:28.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.701+0000 7f5c58f9d640 10 monclient: get_monmap_and_config 2024-03-25T18:18:28.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.701+0000 7f5c58f9d640 10 monclient: build_initial_monmap 2024-03-25T18:18:28.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.701+0000 7f5c58f9d640 10 monclient: monmap: 2024-03-25T18:18:28.702 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:18:28.702 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:28.702 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:18:28.702449+0000 2024-03-25T18:18:28.703 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:18:28.702449+0000 2024-03-25T18:18:28.703 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:28.703 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:28.703 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:18:28.703 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:18:28.703 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:18:28.703 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:28.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.702+0000 7f5c58f9d640 1 Processor -- start 2024-03-25T18:18:28.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.702+0000 7f5c58f9d640 1 -- start start 2024-03-25T18:18:28.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.702+0000 7f5c58f9d640 10 monclient: init 2024-03-25T18:18:28.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.703+0000 7f5c58f9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:28.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.703+0000 7f5c58f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:28.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.703+0000 7f5c58f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f5c540b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T18:18:28.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.703+0000 7f5c58f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f5c540b1730 addr v1:172.21.15.63:6789/0 2024-03-25T18:18:28.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.703+0000 7f5c58f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f5c5414be90 addr v1:172.21.15.81:6789/0 2024-03-25T18:18:28.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.703+0000 7f5c58f9d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c540af940 con 0x7f5c540b1730 2024-03-25T18:18:28.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.703+0000 7f5c58f9d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c540b0a70 con 0x7f5c5414be90 2024-03-25T18:18:28.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.703+0000 7f5c58f9d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c540b0d50 con 0x7f5c540b4d60 2024-03-25T18:18:28.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.703+0000 7f5c58f9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:28.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.703+0000 7f5c58f9d640 10 monclient(hunting): authenticate will time out at 7802.889611s 2024-03-25T18:18:28.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.715+0000 7f5c5259c640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f5c540b4d60 0x7f5c5413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:48030/0 (socket says 172.21.15.63:48030) 2024-03-25T18:18:28.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.715+0000 7f5c5259c640 1 -- 172.21.15.63:0/949703847 learned_addr learned my addr 172.21.15.63:0/949703847 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:18:28.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.732+0000 7f5c51d9b640 1 -- 172.21.15.63:0/949703847 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544167449 0 0) 0x7f5c540b0d50 con 0x7f5c540b4d60 2024-03-25T18:18:28.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.732+0000 7f5c51d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:28.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.732+0000 7f5c51d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:28.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.732+0000 7f5c51d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:28.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.732+0000 7f5c51d9b640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:18:28.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.732+0000 7f5c51d9b640 1 -- 172.21.15.63:0/949703847 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c40003520 con 0x7f5c540b4d60 2024-03-25T18:18:28.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.744+0000 7f5c51d9b640 1 -- 172.21.15.63:0/949703847 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044267894 0 0) 0x7f5c540b0a70 con 0x7f5c5414be90 2024-03-25T18:18:28.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.744+0000 7f5c51d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:28.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.744+0000 7f5c51d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:28.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.744+0000 7f5c51d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:28.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.744+0000 7f5c51d9b640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:18:28.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.744+0000 7f5c51d9b640 1 -- 172.21.15.63:0/949703847 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c540b0d50 con 0x7f5c5414be90 2024-03-25T18:18:28.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.745+0000 7f5c51d9b640 1 -- 172.21.15.63:0/949703847 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066319939 0 0) 0x7f5c540af940 con 0x7f5c540b1730 2024-03-25T18:18:28.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.745+0000 7f5c51d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:28.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.745+0000 7f5c51d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:28.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.745+0000 7f5c51d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:28.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.745+0000 7f5c51d9b640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:18:28.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.745+0000 7f5c51d9b640 1 -- 172.21.15.63:0/949703847 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c540b0a70 con 0x7f5c540b1730 2024-03-25T18:18:28.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c51d9b640 1 -- 172.21.15.63:0/949703847 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 440385179 0 0) 0x7f5c540b0a70 con 0x7f5c540b1730 2024-03-25T18:18:28.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c51d9b640 1 -- 172.21.15.63:0/949703847 >> v1:172.21.15.63:6790/0 conn(0x7f5c540b4d60 legacy=0x7f5c5413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:28.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c51d9b640 1 -- 172.21.15.63:0/949703847 >> v1:172.21.15.81:6789/0 conn(0x7f5c5414be90 legacy=0x7f5c5414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:28.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c51d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:18:28.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c51d9b640 1 monclient: found mon.noname-a 2024-03-25T18:18:28.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c51d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:28.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c51d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:18:28.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c51d9b640 1 -- 172.21.15.63:0/949703847 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c540b1030 con 0x7f5c540b1730 2024-03-25T18:18:28.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c51d9b640 10 monclient: _finish_auth 0 2024-03-25T18:18:28.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c51d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:18:28.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c51d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:28.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c51d9b640 1 -- 172.21.15.63:0/949703847 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5c3c0015b0 con 0x7f5c540b1730 2024-03-25T18:18:28.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c51d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:28.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c51d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:28.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c51d9b640 10 monclient: dump: 2024-03-25T18:18:28.759 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:28.759 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:28.759 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:28.759 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:28.759 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:28.759 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:28.759 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:28.759 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:28.759 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:28.759 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:28.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c58f9d640 5 monclient: authenticate success, global_id 4467 2024-03-25T18:18:28.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.756+0000 7f5c58f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:28.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.765+0000 7f5c51d9b640 1 -- 172.21.15.63:0/949703847 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c3c001710 con 0x7f5c540b1730 2024-03-25T18:18:28.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.765+0000 7f5c51d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:28.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.765+0000 7f5c51d9b640 1 -- 172.21.15.63:0/949703847 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5c3c001b60 con 0x7f5c540b1730 2024-03-25T18:18:28.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.765+0000 7f5c58f9d640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:28.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.765+0000 7f5c58f9d640 4 set_mon_vals no callback set 2024-03-25T18:18:28.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.765+0000 7f5c51d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:28.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.765+0000 7f5c51d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:28.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.765+0000 7f5c51d9b640 10 monclient: dump: 2024-03-25T18:18:28.767 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:28.767 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:28.767 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:28.767 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:28.767 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:28.767 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:28.767 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:28.767 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:28.767 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:28.767 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:28.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.766+0000 7f5c58f9d640 10 monclient: shutdown 2024-03-25T18:18:28.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.766+0000 7f5c58f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:28.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.766+0000 7f5c58f9d640 1 -- 172.21.15.63:0/949703847 >> v1:172.21.15.63:6789/0 conn(0x7f5c540b1730 legacy=0x7f5c540b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:28.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.766+0000 7f5c58f9d640 1 -- 172.21.15.63:0/949703847 shutdown_connections 2024-03-25T18:18:28.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.766+0000 7f5c58f9d640 1 -- 172.21.15.63:0/949703847 >> 172.21.15.63:0/949703847 conn(0x7f5c540617a0 msgr2=0x7f5c54061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:28.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.766+0000 7f5c58f9d640 1 -- 172.21.15.63:0/949703847 shutdown_connections 2024-03-25T18:18:28.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.766+0000 7f5c58f9d640 1 -- 172.21.15.63:0/949703847 wait complete. 2024-03-25T18:18:28.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.767+0000 7f5c58f9d640 10 monclient: build_initial_monmap 2024-03-25T18:18:28.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.767+0000 7f5c58f9d640 10 monclient: monmap: 2024-03-25T18:18:28.770 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:18:28.770 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:28.770 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:18:28.770 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:18:28.770 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:28.770 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:18:28.770 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:18:28.770 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:18:28.771 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:18:28.771 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:28.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.767+0000 7f5c58f9d640 1 librados: starting msgr at 2024-03-25T18:18:28.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.767+0000 7f5c58f9d640 1 librados: starting objecter 2024-03-25T18:18:28.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.767+0000 7f5c58f9d640 1 Processor -- start 2024-03-25T18:18:28.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.767+0000 7f5c58f9d640 1 -- start start 2024-03-25T18:18:28.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.767+0000 7f5c58f9d640 1 librados: setting wanted keys 2024-03-25T18:18:28.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.767+0000 7f5c58f9d640 1 librados: calling monclient init 2024-03-25T18:18:28.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.767+0000 7f5c58f9d640 10 monclient: init 2024-03-25T18:18:28.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.768+0000 7f5c58f9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:28.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.768+0000 7f5c58f9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:28.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.768+0000 7f5c58f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f5c540b1730 addr v1:172.21.15.63:6789/0 2024-03-25T18:18:28.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.768+0000 7f5c58f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f5c540b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T18:18:28.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.768+0000 7f5c58f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f5c5414be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:18:28.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.768+0000 7f5c58f9d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5413bc10 con 0x7f5c540b1730 2024-03-25T18:18:28.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.768+0000 7f5c58f9d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c541d8550 con 0x7f5c540b4d60 2024-03-25T18:18:28.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.768+0000 7f5c58f9d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c541d86e0 con 0x7f5c5414be90 2024-03-25T18:18:28.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.768+0000 7f5c58f9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:28.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.768+0000 7f5c58f9d640 10 monclient(hunting): authenticate will time out at 7802.954810s 2024-03-25T18:18:28.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.788+0000 7f5c4bfff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f5c540b4d60 0x7f5c541d7e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:37326/0 (socket says 172.21.15.63:37326) 2024-03-25T18:18:28.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.789+0000 7f5c4bfff640 1 -- 172.21.15.63:0/1944432250 learned_addr learned my addr 172.21.15.63:0/1944432250 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:18:28.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.792+0000 7f5c4affd640 10 client.?.objecter ms_handle_connect 0x7f5c540b4d60 2024-03-25T18:18:28.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.792+0000 7f5c4affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:28.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.793+0000 7f5c4affd640 10 client.?.objecter ms_handle_connect 0x7f5c5414be90 2024-03-25T18:18:28.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.793+0000 7f5c4affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:28.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.799+0000 7f5c4affd640 10 client.?.objecter ms_handle_connect 0x7f5c540b1730 2024-03-25T18:18:28.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.799+0000 7f5c4affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:28.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.800+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813944066 0 0) 0x7f5c541d86e0 con 0x7f5c5414be90 2024-03-25T18:18:28.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.801+0000 7f5c4affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:28.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.801+0000 7f5c4affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:28.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.801+0000 7f5c4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:28.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.801+0000 7f5c4affd640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:18:28.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.801+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c34004eb0 con 0x7f5c5414be90 2024-03-25T18:18:28.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.802+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586291352 0 0) 0x7f5c541d8550 con 0x7f5c540b4d60 2024-03-25T18:18:28.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.802+0000 7f5c4affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:28.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.802+0000 7f5c4affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:28.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.802+0000 7f5c4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:28.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.802+0000 7f5c4affd640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:18:28.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.802+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c541d86e0 con 0x7f5c540b4d60 2024-03-25T18:18:28.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.816+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2403943356 0 0) 0x7f5c34004eb0 con 0x7f5c5414be90 2024-03-25T18:18:28.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.816+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c541d8550 con 0x7f5c5414be90 2024-03-25T18:18:28.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.816+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5c4c001bd0 con 0x7f5c5414be90 2024-03-25T18:18:28.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.816+0000 7f5c4affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:28.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.816+0000 7f5c4affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:28.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.816+0000 7f5c4affd640 10 monclient(hunting): dump: 2024-03-25T18:18:28.817 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:28.817 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:28.818 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:28.818 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:28.818 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:28.818 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:28.818 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:28.818 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:28.818 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:28.818 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:28.818 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.816+0000 7f5c4affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:28.820 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.819+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582358129 0 0) 0x7f5c5413bc10 con 0x7f5c540b1730 2024-03-25T18:18:28.820 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.819+0000 7f5c4affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:28.820 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.819+0000 7f5c4affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:28.820 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.819+0000 7f5c4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:28.820 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.819+0000 7f5c4affd640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:18:28.820 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.819+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c34004eb0 con 0x7f5c540b1730 2024-03-25T18:18:28.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2141987338 0 0) 0x7f5c541d8550 con 0x7f5c5414be90 2024-03-25T18:18:28.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 >> v1:172.21.15.81:6789/0 conn(0x7f5c540b4d60 legacy=0x7f5c541d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:28.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 >> v1:172.21.15.63:6789/0 conn(0x7f5c540b1730 legacy=0x7f5c5413a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:28.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c4affd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:28.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c4affd640 1 monclient: found mon.c 2024-03-25T18:18:28.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:28.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c4affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:18:28.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5413d0f0 con 0x7f5c5414be90 2024-03-25T18:18:28.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c4affd640 10 monclient: _finish_auth 0 2024-03-25T18:18:28.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c4affd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:28.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:28.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c58f9d640 5 monclient: authenticate success, global_id 4478 2024-03-25T18:18:28.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c58f9d640 10 monclient: _renew_subs 2024-03-25T18:18:28.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c58f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:18:28.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c58f9d640 1 -- 172.21.15.63:0/1944432250 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5413d340 con 0x7f5c5414be90 2024-03-25T18:18:28.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c58f9d640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:28.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c58f9d640 10 monclient: _renew_subs 2024-03-25T18:18:28.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c58f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:18:28.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c58f9d640 1 -- 172.21.15.63:0/1944432250 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c541335f0 con 0x7f5c5414be90 2024-03-25T18:18:28.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.823+0000 7f5c58f9d640 1 librados: init done 2024-03-25T18:18:28.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.824+0000 7f5c58f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:28.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.824+0000 7f5c58f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:28.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.824+0000 7f5c58f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:18:28.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:28.824+0000 7f5c58f9d640 1 -- 172.21.15.63:0/1944432250 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c540627b0 con 0x7f5c5414be90 2024-03-25T18:18:29.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.147+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c4c0020c0 con 0x7f5c5414be90 2024-03-25T18:18:29.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.147+0000 7f5c4affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:29.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.147+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5c4c003b30 con 0x7f5c5414be90 2024-03-25T18:18:29.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.147+0000 7f5c4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:29.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.147+0000 7f5c4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:29.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.147+0000 7f5c4affd640 10 monclient: dump: 2024-03-25T18:18:29.149 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:29.149 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:29.149 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:29.149 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:29.149 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:29.149 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:29.149 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:29.149 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:29.149 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:29.150 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:29.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.147+0000 7f5c5159a640 4 set_mon_vals no callback set 2024-03-25T18:18:29.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.148+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f5c4c019b00 con 0x7f5c5414be90 2024-03-25T18:18:29.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.148+0000 7f5c5259c640 10 monclient: get_auth_request con 0x7f5c3406ddb0 auth_method 0 2024-03-25T18:18:29.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.149+0000 7f5c4affd640 10 client.4478.objecter ms_handle_connect 0x7f5c3406ddb0 2024-03-25T18:18:29.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.155+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f5c4c053790 con 0x7f5c5414be90 2024-03-25T18:18:29.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.155+0000 7f5c4affd640 10 client.4478.objecter ms_dispatch 0x7f5c54000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:29.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.155+0000 7f5c4affd640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:29.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.155+0000 7f5c4affd640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:29.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.155+0000 7f5c4affd640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-25T18:18:29.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.155+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5c4c04f0d0 con 0x7f5c5414be90 2024-03-25T18:18:29.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.155+0000 7f5c4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:29.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.155+0000 7f5c4affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:29.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.399+0000 7f5c58f9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:29.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.399+0000 7f5c58f9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:29.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.399+0000 7f5c58f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:18:29.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.399+0000 7f5c58f9d640 1 -- 172.21.15.63:0/1944432250 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c54069f40 con 0x7f5c5414be90 2024-03-25T18:18:29.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.522+0000 7f5c4affd640 1 -- 172.21.15.63:0/1944432250 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f5c4c0511d0 con 0x7f5c5414be90 2024-03-25T18:18:29.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.522+0000 7f5c4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:29.523 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.522+0000 7f5c4affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:29.524 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:18:29.524 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:29.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.524+0000 7f5c58f9d640 10 librados: watch_flush enter 2024-03-25T18:18:29.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.524+0000 7f5c58f9d640 10 librados: watch_flush exit 2024-03-25T18:18:29.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.524+0000 7f5c58f9d640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-25T18:18:29.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.524+0000 7f5c58f9d640 10 client.4478.objecter successfully canceled tick 2024-03-25T18:18:29.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.524+0000 7f5c58f9d640 1 -- 172.21.15.63:0/1944432250 >> v1:172.21.15.81:6800/3640340873 conn(0x7f5c3406ddb0 legacy=0x7f5c34070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:29.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.524+0000 7f5c58f9d640 10 monclient: shutdown 2024-03-25T18:18:29.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.524+0000 7f5c58f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:29.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.524+0000 7f5c58f9d640 1 -- 172.21.15.63:0/1944432250 >> v1:172.21.15.63:6790/0 conn(0x7f5c5414be90 legacy=0x7f5c54132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:29.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.524+0000 7f5c58f9d640 1 -- 172.21.15.63:0/1944432250 shutdown_connections 2024-03-25T18:18:29.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.524+0000 7f5c58f9d640 1 -- 172.21.15.63:0/1944432250 >> 172.21.15.63:0/1944432250 conn(0x7f5c540617a0 msgr2=0x7f5c540b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:29.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.524+0000 7f5c58f9d640 1 -- 172.21.15.63:0/1944432250 shutdown_connections 2024-03-25T18:18:29.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.524+0000 7f5c58f9d640 1 -- 172.21.15.63:0/1944432250 wait complete. 2024-03-25T18:18:29.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:29.524+0000 7f5c58f9d640 1 librados: shutdown 2024-03-25T18:18:29.537 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:36.343 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:36.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.455+0000 7fecacf82640 10 monclient: get_monmap_and_config 2024-03-25T18:18:36.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.455+0000 7fecacf82640 10 monclient: build_initial_monmap 2024-03-25T18:18:36.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.456+0000 7fecacf82640 10 monclient: monmap: 2024-03-25T18:18:36.457 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:18:36.457 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:36.457 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:18:36.457097+0000 2024-03-25T18:18:36.457 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:18:36.457097+0000 2024-03-25T18:18:36.457 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:36.457 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:36.457 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:18:36.457 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:18:36.457 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:18:36.457 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:36.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.456+0000 7fecacf82640 1 Processor -- start 2024-03-25T18:18:36.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.456+0000 7fecacf82640 1 -- start start 2024-03-25T18:18:36.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.456+0000 7fecacf82640 10 monclient: init 2024-03-25T18:18:36.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.457+0000 7fecacf82640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:36.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.457+0000 7fecacf82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:36.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.457+0000 7fecacf82640 10 monclient(hunting): picked mon.noname-b con 0x7feca80b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T18:18:36.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.457+0000 7fecacf82640 10 monclient(hunting): picked mon.noname-c con 0x7feca80b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:18:36.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.457+0000 7fecacf82640 10 monclient(hunting): picked mon.noname-a con 0x7feca814be90 addr v1:172.21.15.63:6789/0 2024-03-25T18:18:36.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.457+0000 7fecacf82640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca80af940 con 0x7feca814be90 2024-03-25T18:18:36.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.457+0000 7fecacf82640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca80b0a70 con 0x7feca80b1730 2024-03-25T18:18:36.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.457+0000 7fecacf82640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca80b0d50 con 0x7feca80b4d60 2024-03-25T18:18:36.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.457+0000 7fecacf82640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:36.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.457+0000 7fecacf82640 10 monclient(hunting): authenticate will time out at 7810.644239s 2024-03-25T18:18:36.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.461+0000 7fec9ffff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7feca80b1730 0x7feca80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:42264/0 (socket says 172.21.15.63:42264) 2024-03-25T18:18:36.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.461+0000 7fec9ffff640 1 -- 172.21.15.63:0/2712057470 learned_addr learned my addr 172.21.15.63:0/2712057470 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:18:36.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.472+0000 7feca5d9b640 1 -- 172.21.15.63:0/2712057470 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080867449 0 0) 0x7feca80b0a70 con 0x7feca80b1730 2024-03-25T18:18:36.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.472+0000 7feca5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:36.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.472+0000 7feca5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:36.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.472+0000 7feca5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:36.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.472+0000 7feca5d9b640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:18:36.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.472+0000 7feca5d9b640 1 -- 172.21.15.63:0/2712057470 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec94003520 con 0x7feca80b1730 2024-03-25T18:18:36.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.482+0000 7feca5d9b640 1 -- 172.21.15.63:0/2712057470 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964925361 0 0) 0x7feca80b0d50 con 0x7feca80b4d60 2024-03-25T18:18:36.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.482+0000 7feca5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:36.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.482+0000 7feca5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:36.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.482+0000 7feca5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:36.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.482+0000 7feca5d9b640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:18:36.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.482+0000 7feca5d9b640 1 -- 172.21.15.63:0/2712057470 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca80b0a70 con 0x7feca80b4d60 2024-03-25T18:18:36.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7feca5d9b640 1 -- 172.21.15.63:0/2712057470 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3867597802 0 0) 0x7fec94003520 con 0x7feca80b1730 2024-03-25T18:18:36.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7feca5d9b640 1 -- 172.21.15.63:0/2712057470 >> v1:172.21.15.63:6790/0 conn(0x7feca80b4d60 legacy=0x7feca813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:36.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7feca5d9b640 1 -- 172.21.15.63:0/2712057470 >> v1:172.21.15.63:6789/0 conn(0x7feca814be90 legacy=0x7feca814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:36.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7feca5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:18:36.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7feca5d9b640 1 monclient: found mon.noname-c 2024-03-25T18:18:36.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7feca5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:36.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7feca5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:18:36.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7feca5d9b640 1 -- 172.21.15.63:0/2712057470 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca80b1030 con 0x7feca80b1730 2024-03-25T18:18:36.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7feca5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:18:36.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7feca5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:18:36.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7feca5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:36.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7feca5d9b640 1 -- 172.21.15.63:0/2712057470 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fec90001690 con 0x7feca80b1730 2024-03-25T18:18:36.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7feca5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:36.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7feca5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:36.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7feca5d9b640 10 monclient: dump: 2024-03-25T18:18:36.488 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:36.488 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:36.488 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:36.488 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:36.488 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:36.488 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:36.488 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:36.488 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:36.488 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:36.489 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:36.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7fecacf82640 5 monclient: authenticate success, global_id 4501 2024-03-25T18:18:36.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.485+0000 7fecacf82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:36.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.496+0000 7feca5d9b640 1 -- 172.21.15.63:0/2712057470 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec900017f0 con 0x7feca80b1730 2024-03-25T18:18:36.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.496+0000 7feca5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:36.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.496+0000 7feca5d9b640 1 -- 172.21.15.63:0/2712057470 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fec90001c00 con 0x7feca80b1730 2024-03-25T18:18:36.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.496+0000 7fecacf82640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:36.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.496+0000 7fecacf82640 4 set_mon_vals no callback set 2024-03-25T18:18:36.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.496+0000 7feca5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:36.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.496+0000 7feca5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:36.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.496+0000 7feca5d9b640 10 monclient: dump: 2024-03-25T18:18:36.497 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:36.498 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:36.498 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:36.498 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:36.498 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:36.498 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:36.498 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:36.498 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:36.498 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:36.498 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:36.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.496+0000 7fecacf82640 10 monclient: shutdown 2024-03-25T18:18:36.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.496+0000 7fecacf82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:36.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.496+0000 7fecacf82640 1 -- 172.21.15.63:0/2712057470 >> v1:172.21.15.81:6789/0 conn(0x7feca80b1730 legacy=0x7feca80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:36.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.497+0000 7fecacf82640 1 -- 172.21.15.63:0/2712057470 shutdown_connections 2024-03-25T18:18:36.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.497+0000 7fecacf82640 1 -- 172.21.15.63:0/2712057470 >> 172.21.15.63:0/2712057470 conn(0x7feca80617a0 msgr2=0x7feca8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:36.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.497+0000 7fecacf82640 1 -- 172.21.15.63:0/2712057470 shutdown_connections 2024-03-25T18:18:36.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.497+0000 7fecacf82640 1 -- 172.21.15.63:0/2712057470 wait complete. 2024-03-25T18:18:36.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.498+0000 7fecacf82640 10 monclient: build_initial_monmap 2024-03-25T18:18:36.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.498+0000 7fecacf82640 10 monclient: monmap: 2024-03-25T18:18:36.502 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:18:36.503 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:36.503 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:18:36.503 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:18:36.503 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:36.503 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:18:36.503 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:18:36.503 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:18:36.503 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:18:36.503 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:36.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.498+0000 7fecacf82640 1 librados: starting msgr at 2024-03-25T18:18:36.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.498+0000 7fecacf82640 1 librados: starting objecter 2024-03-25T18:18:36.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.498+0000 7fecacf82640 1 Processor -- start 2024-03-25T18:18:36.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.498+0000 7fecacf82640 1 -- start start 2024-03-25T18:18:36.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.498+0000 7fecacf82640 1 librados: setting wanted keys 2024-03-25T18:18:36.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.498+0000 7fecacf82640 1 librados: calling monclient init 2024-03-25T18:18:36.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.498+0000 7fecacf82640 10 monclient: init 2024-03-25T18:18:36.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.499+0000 7fecacf82640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:36.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.499+0000 7fecacf82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:36.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.499+0000 7fecacf82640 10 monclient(hunting): picked mon.noname-c con 0x7feca80b1730 addr v1:172.21.15.63:6790/0 2024-03-25T18:18:36.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.499+0000 7fecacf82640 10 monclient(hunting): picked mon.noname-b con 0x7feca80b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T18:18:36.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.499+0000 7fecacf82640 10 monclient(hunting): picked mon.noname-a con 0x7feca814be90 addr v1:172.21.15.63:6789/0 2024-03-25T18:18:36.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.499+0000 7fecacf82640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca8135810 con 0x7feca814be90 2024-03-25T18:18:36.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.499+0000 7fecacf82640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca8144c10 con 0x7feca80b4d60 2024-03-25T18:18:36.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.499+0000 7fecacf82640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca8144ef0 con 0x7feca80b1730 2024-03-25T18:18:36.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.499+0000 7fecacf82640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:36.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.499+0000 7fecacf82640 10 monclient(hunting): authenticate will time out at 7810.685507s 2024-03-25T18:18:36.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.512+0000 7fec9ffff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7feca80b4d60 0x7feca81349f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:42336/0 (socket says 172.21.15.63:42336) 2024-03-25T18:18:36.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.512+0000 7fec9ffff640 1 -- 172.21.15.63:0/570299390 learned_addr learned my addr 172.21.15.63:0/570299390 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:18:36.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.514+0000 7fec9effd640 10 client.?.objecter ms_handle_connect 0x7feca80b4d60 2024-03-25T18:18:36.515 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.514+0000 7fec9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:36.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.520+0000 7fec9effd640 10 client.?.objecter ms_handle_connect 0x7feca814be90 2024-03-25T18:18:36.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.520+0000 7fec9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:36.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.520+0000 7fec9effd640 10 client.?.objecter ms_handle_connect 0x7feca80b1730 2024-03-25T18:18:36.521 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.520+0000 7fec9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:36.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.533+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009551092 0 0) 0x7feca8144c10 con 0x7feca80b4d60 2024-03-25T18:18:36.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.533+0000 7fec9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:36.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.533+0000 7fec9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:36.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.533+0000 7fec9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:36.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.533+0000 7fec9effd640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:18:36.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.533+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec88004eb0 con 0x7feca80b4d60 2024-03-25T18:18:36.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.537+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50197323 0 0) 0x7feca8135810 con 0x7feca814be90 2024-03-25T18:18:36.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.537+0000 7fec9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:36.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.537+0000 7fec9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:36.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.537+0000 7fec9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:36.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.537+0000 7fec9effd640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:18:36.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.537+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca8144c10 con 0x7feca814be90 2024-03-25T18:18:36.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.555+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2531497379 0 0) 0x7fec88004eb0 con 0x7feca80b4d60 2024-03-25T18:18:36.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.555+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feca8135810 con 0x7feca80b4d60 2024-03-25T18:18:36.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.555+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fec90001bc0 con 0x7feca80b4d60 2024-03-25T18:18:36.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.555+0000 7fec9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:36.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.555+0000 7fec9effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:36.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.555+0000 7fec9effd640 10 monclient(hunting): dump: 2024-03-25T18:18:36.557 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:36.557 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:36.557 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:36.557 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:36.557 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:36.557 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:36.557 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:36.557 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:36.557 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:36.557 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:36.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.555+0000 7fec9effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:36.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.558+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124799870 0 0) 0x7feca8144ef0 con 0x7feca80b1730 2024-03-25T18:18:36.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.558+0000 7fec9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:36.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.558+0000 7fec9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:36.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.558+0000 7fec9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:36.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.558+0000 7fec9effd640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:18:36.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.558+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec88004eb0 con 0x7feca80b1730 2024-03-25T18:18:36.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.561+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3487525219 0 0) 0x7feca8144c10 con 0x7feca814be90 2024-03-25T18:18:36.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.561+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feca8144ef0 con 0x7feca814be90 2024-03-25T18:18:36.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.561+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7feca0001b00 con 0x7feca814be90 2024-03-25T18:18:36.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.561+0000 7fec9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:36.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.561+0000 7fec9effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:36.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.561+0000 7fec9effd640 10 monclient(hunting): dump: 2024-03-25T18:18:36.563 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:36.563 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:36.563 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:36.563 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:36.563 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:36.563 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:36.563 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:36.563 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:36.563 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:36.563 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:36.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 705220802 0 0) 0x7feca8144ef0 con 0x7feca814be90 2024-03-25T18:18:36.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 >> v1:172.21.15.63:6790/0 conn(0x7feca80b1730 legacy=0x7feca81d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:36.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 >> v1:172.21.15.81:6789/0 conn(0x7feca80b4d60 legacy=0x7feca81349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:36.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fec9effd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:36.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fec9effd640 1 monclient: found mon.a 2024-03-25T18:18:36.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fec9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:36.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fec9effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:18:36.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca8140c10 con 0x7feca814be90 2024-03-25T18:18:36.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fec9effd640 10 monclient: _finish_auth 0 2024-03-25T18:18:36.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fec9effd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:36.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fec9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:36.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fecacf82640 5 monclient: authenticate success, global_id 4476 2024-03-25T18:18:36.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fecacf82640 10 monclient: _renew_subs 2024-03-25T18:18:36.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fecacf82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:18:36.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fecacf82640 1 -- 172.21.15.63:0/570299390 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca8137d50 con 0x7feca814be90 2024-03-25T18:18:36.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fecacf82640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:36.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fecacf82640 10 monclient: _renew_subs 2024-03-25T18:18:36.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fecacf82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:18:36.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fecacf82640 1 -- 172.21.15.63:0/570299390 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feca81407d0 con 0x7feca814be90 2024-03-25T18:18:36.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.576+0000 7fecacf82640 1 librados: init done 2024-03-25T18:18:36.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.578+0000 7fecacf82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:36.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.578+0000 7fecacf82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:36.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.578+0000 7fecacf82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:18:36.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.578+0000 7fecacf82640 1 -- 172.21.15.63:0/570299390 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca8141e30 con 0x7feca814be90 2024-03-25T18:18:36.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.907+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feca0001fb0 con 0x7feca814be90 2024-03-25T18:18:36.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.907+0000 7fec9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:36.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.907+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7feca0003b10 con 0x7feca814be90 2024-03-25T18:18:36.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.907+0000 7fec9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:36.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.907+0000 7fec9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:36.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.907+0000 7fec9effd640 10 monclient: dump: 2024-03-25T18:18:36.909 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:36.909 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:36.909 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:36.909 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:36.909 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:36.909 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:36.909 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:36.909 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:36.909 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:36.909 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:36.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.907+0000 7feca559a640 4 set_mon_vals no callback set 2024-03-25T18:18:36.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.907+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7feca0003d30 con 0x7feca814be90 2024-03-25T18:18:36.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.908+0000 7feca659c640 10 monclient: get_auth_request con 0x7fec8806df70 auth_method 0 2024-03-25T18:18:36.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.908+0000 7fec9effd640 10 client.4476.objecter ms_handle_connect 0x7fec8806df70 2024-03-25T18:18:36.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.916+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7feca00536d0 con 0x7feca814be90 2024-03-25T18:18:36.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.916+0000 7fec9effd640 10 client.4476.objecter ms_dispatch 0x7feca8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:36.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.916+0000 7fec9effd640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:36.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.916+0000 7fec9effd640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:36.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.916+0000 7fec9effd640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-25T18:18:36.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.916+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feca004f1e0 con 0x7feca814be90 2024-03-25T18:18:36.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.916+0000 7fec9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:36.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:36.916+0000 7fec9effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:37.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.134+0000 7fecacf82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:37.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.134+0000 7fecacf82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:37.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.134+0000 7fecacf82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:18:37.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.134+0000 7fecacf82640 1 -- 172.21.15.63:0/570299390 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feca81380a0 con 0x7feca814be90 2024-03-25T18:18:37.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.177+0000 7fec9effd640 1 -- 172.21.15.63:0/570299390 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7feca81380a0 con 0x7feca814be90 2024-03-25T18:18:37.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.177+0000 7fec9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:37.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.177+0000 7fec9effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:37.178 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:18:37.178 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:37.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.179+0000 7fecacf82640 10 librados: watch_flush enter 2024-03-25T18:18:37.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.179+0000 7fecacf82640 10 librados: watch_flush exit 2024-03-25T18:18:37.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.179+0000 7fecacf82640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-25T18:18:37.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.179+0000 7fecacf82640 10 client.4476.objecter successfully canceled tick 2024-03-25T18:18:37.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.179+0000 7fecacf82640 1 -- 172.21.15.63:0/570299390 >> v1:172.21.15.81:6800/3640340873 conn(0x7fec8806df70 legacy=0x7fec88070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.179+0000 7fecacf82640 10 monclient: shutdown 2024-03-25T18:18:37.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.179+0000 7fecacf82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:37.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.179+0000 7fecacf82640 1 -- 172.21.15.63:0/570299390 >> v1:172.21.15.63:6789/0 conn(0x7feca814be90 legacy=0x7feca8135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.179+0000 7fecacf82640 1 -- 172.21.15.63:0/570299390 shutdown_connections 2024-03-25T18:18:37.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.179+0000 7fecacf82640 1 -- 172.21.15.63:0/570299390 >> 172.21.15.63:0/570299390 conn(0x7feca80617a0 msgr2=0x7feca80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:37.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.180+0000 7fecacf82640 1 -- 172.21.15.63:0/570299390 shutdown_connections 2024-03-25T18:18:37.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.180+0000 7fecacf82640 1 -- 172.21.15.63:0/570299390 wait complete. 2024-03-25T18:18:37.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:37.180+0000 7fecacf82640 1 librados: shutdown 2024-03-25T18:18:37.193 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:39.146 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:20.838 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:39.154 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:20.846 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:39.181 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:20.873 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:39.206 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:06:20.866 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:39.214 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:06:20.874 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:39.220 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:20.912 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:39.235 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:20.928 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:39.249 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:06:20.909 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:39.272 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:20.964 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:43.999 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:44.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.117+0000 7ff489b6b640 10 monclient: get_monmap_and_config 2024-03-25T18:18:44.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.117+0000 7ff489b6b640 10 monclient: build_initial_monmap 2024-03-25T18:18:44.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.117+0000 7ff489b6b640 10 monclient: monmap: 2024-03-25T18:18:44.118 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:18:44.118 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:44.118 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:18:44.118466+0000 2024-03-25T18:18:44.119 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:18:44.118466+0000 2024-03-25T18:18:44.119 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:44.119 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:44.119 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:18:44.119 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:18:44.119 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:18:44.119 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:44.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.118+0000 7ff489b6b640 1 Processor -- start 2024-03-25T18:18:44.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.118+0000 7ff489b6b640 1 -- start start 2024-03-25T18:18:44.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.118+0000 7ff489b6b640 10 monclient: init 2024-03-25T18:18:44.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.118+0000 7ff489b6b640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:44.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.118+0000 7ff489b6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:44.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.118+0000 7ff489b6b640 10 monclient(hunting): picked mon.noname-c con 0x7ff4840a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:18:44.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.118+0000 7ff489b6b640 10 monclient(hunting): picked mon.noname-a con 0x7ff48413f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:18:44.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.118+0000 7ff489b6b640 10 monclient(hunting): picked mon.noname-b con 0x7ff4841430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:18:44.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.118+0000 7ff489b6b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff484142520 con 0x7ff48413f710 2024-03-25T18:18:44.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.119+0000 7ff489b6b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4840b28a0 con 0x7ff4840a98f0 2024-03-25T18:18:44.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.119+0000 7ff489b6b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4840b2a30 con 0x7ff4841430c0 2024-03-25T18:18:44.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.119+0000 7ff489b6b640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:44.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.119+0000 7ff489b6b640 10 monclient(hunting): authenticate will time out at 7818.305311s 2024-03-25T18:18:44.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.131+0000 7ff482ffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7ff48413f710 0x7ff4841429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59930/0 (socket says 172.21.15.63:59930) 2024-03-25T18:18:44.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.131+0000 7ff482ffd640 1 -- 172.21.15.63:0/154962465 learned_addr learned my addr 172.21.15.63:0/154962465 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:18:44.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.149+0000 7ff4827fc640 1 -- 172.21.15.63:0/154962465 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423669698 0 0) 0x7ff4840b28a0 con 0x7ff4840a98f0 2024-03-25T18:18:44.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.149+0000 7ff4827fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:44.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.149+0000 7ff4827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:44.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.149+0000 7ff4827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:44.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.149+0000 7ff4827fc640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:18:44.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.149+0000 7ff4827fc640 1 -- 172.21.15.63:0/154962465 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff46c003520 con 0x7ff4840a98f0 2024-03-25T18:18:44.153 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.152+0000 7ff4827fc640 1 -- 172.21.15.63:0/154962465 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95152340 0 0) 0x7ff484142520 con 0x7ff48413f710 2024-03-25T18:18:44.153 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.152+0000 7ff4827fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:44.153 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.152+0000 7ff4827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:44.153 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.152+0000 7ff4827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:44.153 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.152+0000 7ff4827fc640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:18:44.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.152+0000 7ff4827fc640 1 -- 172.21.15.63:0/154962465 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4840b28a0 con 0x7ff48413f710 2024-03-25T18:18:44.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff4827fc640 1 -- 172.21.15.63:0/154962465 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2568842684 0 0) 0x7ff46c003520 con 0x7ff4840a98f0 2024-03-25T18:18:44.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff4827fc640 1 -- 172.21.15.63:0/154962465 >> v1:172.21.15.63:6790/0 conn(0x7ff4841430c0 legacy=0x7ff48414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:44.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff4827fc640 1 -- 172.21.15.63:0/154962465 >> v1:172.21.15.63:6789/0 conn(0x7ff48413f710 legacy=0x7ff4841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:44.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff4827fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:44.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff4827fc640 1 monclient: found mon.noname-c 2024-03-25T18:18:44.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff4827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:44.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff4827fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:18:44.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff4827fc640 1 -- 172.21.15.63:0/154962465 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4840b44c0 con 0x7ff4840a98f0 2024-03-25T18:18:44.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff4827fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:44.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff4827fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:44.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff4827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:44.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff4827fc640 1 -- 172.21.15.63:0/154962465 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ff474001690 con 0x7ff4840a98f0 2024-03-25T18:18:44.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff4827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:44.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff4827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:44.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff4827fc640 10 monclient: dump: 2024-03-25T18:18:44.166 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:44.166 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:44.166 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:44.166 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:44.166 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:44.166 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:44.166 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:44.167 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:44.167 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:44.167 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:44.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff489b6b640 5 monclient: authenticate success, global_id 4507 2024-03-25T18:18:44.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.164+0000 7ff489b6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:44.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.169+0000 7ff4827fc640 1 -- 172.21.15.63:0/154962465 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4740017f0 con 0x7ff4840a98f0 2024-03-25T18:18:44.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.169+0000 7ff4827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:44.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.169+0000 7ff489b6b640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:44.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.169+0000 7ff489b6b640 4 set_mon_vals no callback set 2024-03-25T18:18:44.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.170+0000 7ff489b6b640 10 monclient: shutdown 2024-03-25T18:18:44.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.170+0000 7ff489b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:44.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.170+0000 7ff489b6b640 1 -- 172.21.15.63:0/154962465 >> v1:172.21.15.81:6789/0 conn(0x7ff4840a98f0 legacy=0x7ff4840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:44.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.170+0000 7ff489b6b640 1 -- 172.21.15.63:0/154962465 shutdown_connections 2024-03-25T18:18:44.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.170+0000 7ff489b6b640 1 -- 172.21.15.63:0/154962465 >> 172.21.15.63:0/154962465 conn(0x7ff4840b7ea0 msgr2=0x7ff4840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:44.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.170+0000 7ff489b6b640 1 -- 172.21.15.63:0/154962465 shutdown_connections 2024-03-25T18:18:44.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.170+0000 7ff489b6b640 1 -- 172.21.15.63:0/154962465 wait complete. 2024-03-25T18:18:44.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.171+0000 7ff489b6b640 10 monclient: build_initial_monmap 2024-03-25T18:18:44.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.171+0000 7ff489b6b640 10 monclient: monmap: 2024-03-25T18:18:44.172 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:18:44.172 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:44.172 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:18:44.172 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:18:44.172 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:44.172 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:18:44.172 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:18:44.173 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:18:44.173 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:18:44.173 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:44.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.171+0000 7ff489b6b640 1 librados: starting msgr at 2024-03-25T18:18:44.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.171+0000 7ff489b6b640 1 librados: starting objecter 2024-03-25T18:18:44.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.171+0000 7ff489b6b640 1 Processor -- start 2024-03-25T18:18:44.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.171+0000 7ff489b6b640 1 -- start start 2024-03-25T18:18:44.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.171+0000 7ff489b6b640 1 librados: setting wanted keys 2024-03-25T18:18:44.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.171+0000 7ff489b6b640 1 librados: calling monclient init 2024-03-25T18:18:44.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.171+0000 7ff489b6b640 10 monclient: init 2024-03-25T18:18:44.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.172+0000 7ff489b6b640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:44.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.172+0000 7ff489b6b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:44.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.172+0000 7ff489b6b640 10 monclient(hunting): picked mon.noname-a con 0x7ff4840a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:18:44.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.172+0000 7ff489b6b640 10 monclient(hunting): picked mon.noname-b con 0x7ff48413f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:18:44.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.172+0000 7ff489b6b640 10 monclient(hunting): picked mon.noname-c con 0x7ff4841430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:18:44.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.172+0000 7ff489b6b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff484139680 con 0x7ff4840a98f0 2024-03-25T18:18:44.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.172+0000 7ff489b6b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff484138cf0 con 0x7ff48413f710 2024-03-25T18:18:44.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.172+0000 7ff489b6b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff484136620 con 0x7ff4841430c0 2024-03-25T18:18:44.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.173+0000 7ff489b6b640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:44.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.173+0000 7ff489b6b640 10 monclient(hunting): authenticate will time out at 7818.359359s 2024-03-25T18:18:44.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.186+0000 7ff4837fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7ff4840a98f0 0x7ff4841371a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59976/0 (socket says 172.21.15.63:59976) 2024-03-25T18:18:44.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.186+0000 7ff4837fe640 1 -- 172.21.15.63:0/3686786567 learned_addr learned my addr 172.21.15.63:0/3686786567 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:18:44.191 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.190+0000 7ff46bfff640 10 client.?.objecter ms_handle_connect 0x7ff48413f710 2024-03-25T18:18:44.191 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.190+0000 7ff46bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:44.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.193+0000 7ff46bfff640 10 client.?.objecter ms_handle_connect 0x7ff4840a98f0 2024-03-25T18:18:44.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.193+0000 7ff46bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:44.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.195+0000 7ff46bfff640 10 client.?.objecter ms_handle_connect 0x7ff4841430c0 2024-03-25T18:18:44.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.195+0000 7ff46bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:44.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.207+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779963876 0 0) 0x7ff484136620 con 0x7ff4841430c0 2024-03-25T18:18:44.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.207+0000 7ff46bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:44.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.207+0000 7ff46bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:44.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.207+0000 7ff46bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:44.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.207+0000 7ff46bfff640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:18:44.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.207+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff470004eb0 con 0x7ff4841430c0 2024-03-25T18:18:44.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.208+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864236580 0 0) 0x7ff484138cf0 con 0x7ff48413f710 2024-03-25T18:18:44.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.208+0000 7ff46bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:44.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.208+0000 7ff46bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:44.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.208+0000 7ff46bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:44.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.208+0000 7ff46bfff640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:18:44.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.208+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff484136620 con 0x7ff48413f710 2024-03-25T18:18:44.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.212+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522008275 0 0) 0x7ff484139680 con 0x7ff4840a98f0 2024-03-25T18:18:44.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.212+0000 7ff46bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:44.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.212+0000 7ff46bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:44.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.212+0000 7ff46bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:44.214 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.212+0000 7ff46bfff640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:18:44.214 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.212+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff484138cf0 con 0x7ff4840a98f0 2024-03-25T18:18:44.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.218+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 325028913 0 0) 0x7ff470004eb0 con 0x7ff4841430c0 2024-03-25T18:18:44.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.218+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff484139680 con 0x7ff4841430c0 2024-03-25T18:18:44.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.218+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ff47c001b90 con 0x7ff4841430c0 2024-03-25T18:18:44.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.218+0000 7ff46bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:44.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.218+0000 7ff46bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:44.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.218+0000 7ff46bfff640 10 monclient(hunting): dump: 2024-03-25T18:18:44.219 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:44.219 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:44.220 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:44.220 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:44.220 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:44.220 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:44.220 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:44.220 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:44.220 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:44.220 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:44.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.218+0000 7ff46bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:44.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.235+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2490982112 0 0) 0x7ff484138cf0 con 0x7ff4840a98f0 2024-03-25T18:18:44.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.235+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff470004eb0 con 0x7ff4840a98f0 2024-03-25T18:18:44.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.235+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ff474001ca0 con 0x7ff4840a98f0 2024-03-25T18:18:44.237 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.235+0000 7ff46bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:44.237 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.235+0000 7ff46bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:44.237 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.235+0000 7ff46bfff640 10 monclient(hunting): dump: 2024-03-25T18:18:44.237 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:44.237 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:44.237 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:44.237 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:44.237 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:44.237 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:44.237 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:44.237 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:44.237 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:44.237 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:44.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 301181470 0 0) 0x7ff484139680 con 0x7ff4841430c0 2024-03-25T18:18:44.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 >> v1:172.21.15.81:6789/0 conn(0x7ff48413f710 legacy=0x7ff4841dabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:44.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 >> v1:172.21.15.63:6789/0 conn(0x7ff4840a98f0 legacy=0x7ff4841371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:44.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff46bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:44.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff46bfff640 1 monclient: found mon.c 2024-03-25T18:18:44.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff46bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:44.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff46bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:18:44.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4840b1620 con 0x7ff4841430c0 2024-03-25T18:18:44.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff46bfff640 10 monclient: _finish_auth 0 2024-03-25T18:18:44.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff46bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:44.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff46bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:44.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff489b6b640 5 monclient: authenticate success, global_id 4490 2024-03-25T18:18:44.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff489b6b640 10 monclient: _renew_subs 2024-03-25T18:18:44.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff489b6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:18:44.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff489b6b640 1 -- 172.21.15.63:0/3686786567 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff484139110 con 0x7ff4841430c0 2024-03-25T18:18:44.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff489b6b640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:44.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff489b6b640 10 monclient: _renew_subs 2024-03-25T18:18:44.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff489b6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:18:44.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff489b6b640 1 -- 172.21.15.63:0/3686786567 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48413b770 con 0x7ff4841430c0 2024-03-25T18:18:44.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.236+0000 7ff489b6b640 1 librados: init done 2024-03-25T18:18:44.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.237+0000 7ff489b6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:44.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.237+0000 7ff489b6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:44.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.237+0000 7ff489b6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:18:44.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.237+0000 7ff489b6b640 1 -- 172.21.15.63:0/3686786567 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff484139830 con 0x7ff4841430c0 2024-03-25T18:18:44.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.549+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff47c002080 con 0x7ff4841430c0 2024-03-25T18:18:44.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.549+0000 7ff46bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:44.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.549+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ff47c003970 con 0x7ff4841430c0 2024-03-25T18:18:44.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.549+0000 7ff46bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:44.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.549+0000 7ff46bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:44.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.549+0000 7ff46bfff640 10 monclient: dump: 2024-03-25T18:18:44.551 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:44.551 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:44.551 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:44.552 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:44.552 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:44.552 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:44.552 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:44.552 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:44.552 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:44.552 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:44.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.549+0000 7ff481ffb640 4 set_mon_vals no callback set 2024-03-25T18:18:44.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.550+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7ff47c003b90 con 0x7ff4841430c0 2024-03-25T18:18:44.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.551+0000 7ff4837fe640 10 monclient: get_auth_request con 0x7ff47006df70 auth_method 0 2024-03-25T18:18:44.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.551+0000 7ff46bfff640 10 client.4490.objecter ms_handle_connect 0x7ff47006df70 2024-03-25T18:18:44.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.560+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7ff47c0536e0 con 0x7ff4841430c0 2024-03-25T18:18:44.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.560+0000 7ff46bfff640 10 client.4490.objecter ms_dispatch 0x7ff484000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:44.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.560+0000 7ff46bfff640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:44.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.560+0000 7ff46bfff640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:44.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.560+0000 7ff46bfff640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-25T18:18:44.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.560+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff47c04f1f0 con 0x7ff4841430c0 2024-03-25T18:18:44.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.560+0000 7ff46bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:44.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.560+0000 7ff46bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:44.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.801+0000 7ff489b6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:44.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.801+0000 7ff489b6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:44.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.801+0000 7ff489b6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:18:44.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.801+0000 7ff489b6b640 1 -- 172.21.15.63:0/3686786567 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4840627b0 con 0x7ff4841430c0 2024-03-25T18:18:44.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.834+0000 7ff46bfff640 1 -- 172.21.15.63:0/3686786567 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7ff47c051120 con 0x7ff4841430c0 2024-03-25T18:18:44.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.834+0000 7ff46bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:44.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.834+0000 7ff46bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:44.836 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:18:44.836 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:44.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.837+0000 7ff489b6b640 10 librados: watch_flush enter 2024-03-25T18:18:44.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.837+0000 7ff489b6b640 10 librados: watch_flush exit 2024-03-25T18:18:44.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.837+0000 7ff489b6b640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-25T18:18:44.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.837+0000 7ff489b6b640 10 client.4490.objecter successfully canceled tick 2024-03-25T18:18:44.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.837+0000 7ff489b6b640 1 -- 172.21.15.63:0/3686786567 >> v1:172.21.15.81:6800/3640340873 conn(0x7ff47006df70 legacy=0x7ff470070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:44.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.837+0000 7ff489b6b640 10 monclient: shutdown 2024-03-25T18:18:44.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.837+0000 7ff489b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:44.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.837+0000 7ff489b6b640 1 -- 172.21.15.63:0/3686786567 >> v1:172.21.15.63:6790/0 conn(0x7ff4841430c0 legacy=0x7ff4841e2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:44.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.837+0000 7ff489b6b640 1 -- 172.21.15.63:0/3686786567 shutdown_connections 2024-03-25T18:18:44.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.837+0000 7ff489b6b640 1 -- 172.21.15.63:0/3686786567 >> 172.21.15.63:0/3686786567 conn(0x7ff4840b7ea0 msgr2=0x7ff484074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:44.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.837+0000 7ff489b6b640 1 -- 172.21.15.63:0/3686786567 shutdown_connections 2024-03-25T18:18:44.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.837+0000 7ff489b6b640 1 -- 172.21.15.63:0/3686786567 wait complete. 2024-03-25T18:18:44.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:44.837+0000 7ff489b6b640 1 librados: shutdown 2024-03-25T18:18:44.850 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:51.656 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:51.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.773+0000 7f27f274d640 10 monclient: get_monmap_and_config 2024-03-25T18:18:51.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.773+0000 7f27f274d640 10 monclient: build_initial_monmap 2024-03-25T18:18:51.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.773+0000 7f27f274d640 10 monclient: monmap: 2024-03-25T18:18:51.774 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:18:51.774 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:51.774 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:18:51.774447+0000 2024-03-25T18:18:51.774 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:18:51.774447+0000 2024-03-25T18:18:51.774 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:51.775 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:51.775 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:18:51.775 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:18:51.775 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:18:51.775 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:51.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.774+0000 7f27f274d640 1 Processor -- start 2024-03-25T18:18:51.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.774+0000 7f27f274d640 1 -- start start 2024-03-25T18:18:51.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.774+0000 7f27f274d640 10 monclient: init 2024-03-25T18:18:51.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.775+0000 7f27f274d640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:51.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.775+0000 7f27f274d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:51.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.775+0000 7f27f274d640 10 monclient(hunting): picked mon.noname-c con 0x7f27ec0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:18:51.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.775+0000 7f27f274d640 10 monclient(hunting): picked mon.noname-a con 0x7f27ec13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:18:51.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.775+0000 7f27f274d640 10 monclient(hunting): picked mon.noname-b con 0x7f27ec1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:18:51.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.775+0000 7f27f274d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ec142520 con 0x7f27ec13f710 2024-03-25T18:18:51.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.775+0000 7f27f274d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ec0b28a0 con 0x7f27ec0a98f0 2024-03-25T18:18:51.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.775+0000 7f27f274d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ec0b2a30 con 0x7f27ec1430c0 2024-03-25T18:18:51.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.775+0000 7f27f274d640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:51.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.775+0000 7f27f274d640 10 monclient(hunting): authenticate will time out at 7825.961707s 2024-03-25T18:18:51.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.784+0000 7f27eb7fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f27ec13f710 0x7f27ec1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:35990/0 (socket says 172.21.15.63:35990) 2024-03-25T18:18:51.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.784+0000 7f27eb7fe640 1 -- 172.21.15.63:0/3701834951 learned_addr learned my addr 172.21.15.63:0/3701834951 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:18:51.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.803+0000 7f27eaffd640 1 -- 172.21.15.63:0/3701834951 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567520576 0 0) 0x7f27ec0b2a30 con 0x7f27ec1430c0 2024-03-25T18:18:51.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.803+0000 7f27eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:51.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.803+0000 7f27eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:51.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.803+0000 7f27eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:51.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.803+0000 7f27eaffd640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:18:51.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.803+0000 7f27eaffd640 1 -- 172.21.15.63:0/3701834951 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27d4003520 con 0x7f27ec1430c0 2024-03-25T18:18:51.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.805+0000 7f27eaffd640 1 -- 172.21.15.63:0/3701834951 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302730737 0 0) 0x7f27ec142520 con 0x7f27ec13f710 2024-03-25T18:18:51.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.805+0000 7f27eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:51.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.805+0000 7f27eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:51.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.805+0000 7f27eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:51.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.805+0000 7f27eaffd640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:18:51.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.805+0000 7f27eaffd640 1 -- 172.21.15.63:0/3701834951 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27ec0b2a30 con 0x7f27ec13f710 2024-03-25T18:18:51.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.820+0000 7f27eaffd640 1 -- 172.21.15.63:0/3701834951 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 910726345 0 0) 0x7f27ec0b2a30 con 0x7f27ec13f710 2024-03-25T18:18:51.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.821+0000 7f27eaffd640 1 -- 172.21.15.63:0/3701834951 >> v1:172.21.15.63:6790/0 conn(0x7f27ec1430c0 legacy=0x7f27ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:51.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.821+0000 7f27eaffd640 1 -- 172.21.15.63:0/3701834951 >> v1:172.21.15.81:6789/0 conn(0x7f27ec0a98f0 legacy=0x7f27ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:51.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.821+0000 7f27eaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:51.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.821+0000 7f27eaffd640 1 monclient: found mon.noname-a 2024-03-25T18:18:51.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.821+0000 7f27eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:51.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.821+0000 7f27eaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:18:51.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.821+0000 7f27eaffd640 1 -- 172.21.15.63:0/3701834951 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec0b44c0 con 0x7f27ec13f710 2024-03-25T18:18:51.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.821+0000 7f27eaffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:51.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.821+0000 7f27eaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:51.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.821+0000 7f27eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:51.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.821+0000 7f27eaffd640 1 -- 172.21.15.63:0/3701834951 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f27e0001670 con 0x7f27ec13f710 2024-03-25T18:18:51.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.821+0000 7f27eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:51.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.821+0000 7f27eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:51.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.821+0000 7f27eaffd640 10 monclient: dump: 2024-03-25T18:18:51.823 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:51.823 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:51.823 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:51.823 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:51.823 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:51.823 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:51.823 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:51.823 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:51.823 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:51.823 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:51.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.821+0000 7f27f274d640 5 monclient: authenticate success, global_id 4485 2024-03-25T18:18:51.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.821+0000 7f27f274d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:51.853 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.851+0000 7f27eaffd640 1 -- 172.21.15.63:0/3701834951 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27e00017d0 con 0x7f27ec13f710 2024-03-25T18:18:51.853 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.851+0000 7f27eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:51.853 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.851+0000 7f27eaffd640 1 -- 172.21.15.63:0/3701834951 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f27e0001b60 con 0x7f27ec13f710 2024-03-25T18:18:51.853 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.851+0000 7f27f274d640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:51.853 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.851+0000 7f27f274d640 4 set_mon_vals no callback set 2024-03-25T18:18:51.853 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.851+0000 7f27eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:51.853 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.851+0000 7f27eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:51.853 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.851+0000 7f27eaffd640 10 monclient: dump: 2024-03-25T18:18:51.853 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:51.853 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:51.854 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:51.854 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:51.854 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:51.854 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:51.854 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:51.854 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:51.854 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:51.854 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:51.856 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.852+0000 7f27f274d640 10 monclient: shutdown 2024-03-25T18:18:51.856 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.852+0000 7f27f274d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:51.856 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.852+0000 7f27f274d640 1 -- 172.21.15.63:0/3701834951 >> v1:172.21.15.63:6789/0 conn(0x7f27ec13f710 legacy=0x7f27ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:51.856 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.852+0000 7f27f274d640 1 -- 172.21.15.63:0/3701834951 shutdown_connections 2024-03-25T18:18:51.856 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.852+0000 7f27f274d640 1 -- 172.21.15.63:0/3701834951 >> 172.21.15.63:0/3701834951 conn(0x7f27ec0b7ea0 msgr2=0x7f27ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:51.856 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.852+0000 7f27f274d640 1 -- 172.21.15.63:0/3701834951 shutdown_connections 2024-03-25T18:18:51.857 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.852+0000 7f27f274d640 1 -- 172.21.15.63:0/3701834951 wait complete. 2024-03-25T18:18:51.857 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.853+0000 7f27f274d640 10 monclient: build_initial_monmap 2024-03-25T18:18:51.857 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.853+0000 7f27f274d640 10 monclient: monmap: 2024-03-25T18:18:51.857 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:18:51.857 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:51.857 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:18:51.857 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:18:51.857 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:51.857 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:18:51.857 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:18:51.857 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:18:51.857 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:18:51.857 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:51.857 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.853+0000 7f27f274d640 1 librados: starting msgr at 2024-03-25T18:18:51.857 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.853+0000 7f27f274d640 1 librados: starting objecter 2024-03-25T18:18:51.857 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.854+0000 7f27f274d640 1 Processor -- start 2024-03-25T18:18:51.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.854+0000 7f27f274d640 1 -- start start 2024-03-25T18:18:51.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.854+0000 7f27f274d640 1 librados: setting wanted keys 2024-03-25T18:18:51.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.854+0000 7f27f274d640 1 librados: calling monclient init 2024-03-25T18:18:51.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.854+0000 7f27f274d640 10 monclient: init 2024-03-25T18:18:51.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.854+0000 7f27f274d640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:51.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.854+0000 7f27f274d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:51.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.854+0000 7f27f274d640 10 monclient(hunting): picked mon.noname-b con 0x7f27ec0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:18:51.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.854+0000 7f27f274d640 10 monclient(hunting): picked mon.noname-a con 0x7f27ec13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:18:51.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.854+0000 7f27f274d640 10 monclient(hunting): picked mon.noname-c con 0x7f27ec1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:18:51.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.854+0000 7f27f274d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ec0b7c80 con 0x7f27ec13f710 2024-03-25T18:18:51.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.854+0000 7f27f274d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ec14e4f0 con 0x7f27ec0a98f0 2024-03-25T18:18:51.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.855+0000 7f27f274d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ec13caf0 con 0x7f27ec1430c0 2024-03-25T18:18:51.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.855+0000 7f27f274d640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:51.858 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.855+0000 7f27f274d640 10 monclient(hunting): authenticate will time out at 7826.041314s 2024-03-25T18:18:51.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.858+0000 7f27ebfff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f27ec0a98f0 0x7f27ec13c3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:56810/0 (socket says 172.21.15.63:56810) 2024-03-25T18:18:51.859 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.858+0000 7f27ebfff640 1 -- 172.21.15.63:0/3953020800 learned_addr learned my addr 172.21.15.63:0/3953020800 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:18:51.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.861+0000 7f27e8ff9640 10 client.?.objecter ms_handle_connect 0x7f27ec0a98f0 2024-03-25T18:18:51.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.861+0000 7f27e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:51.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.863+0000 7f27e8ff9640 10 client.?.objecter ms_handle_connect 0x7f27ec13f710 2024-03-25T18:18:51.864 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.863+0000 7f27e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:51.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.871+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315496605 0 0) 0x7f27ec14e4f0 con 0x7f27ec0a98f0 2024-03-25T18:18:51.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.871+0000 7f27e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:51.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.871+0000 7f27e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:51.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.871+0000 7f27e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:51.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.871+0000 7f27e8ff9640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:18:51.873 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.871+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27d8004eb0 con 0x7f27ec0a98f0 2024-03-25T18:18:51.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.879+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194987799 0 0) 0x7f27ec0b7c80 con 0x7f27ec13f710 2024-03-25T18:18:51.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.879+0000 7f27e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:51.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.879+0000 7f27e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:51.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.879+0000 7f27e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:51.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.879+0000 7f27e8ff9640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:18:51.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.879+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27ec14e4f0 con 0x7f27ec13f710 2024-03-25T18:18:51.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.886+0000 7f27e8ff9640 10 client.?.objecter ms_handle_connect 0x7f27ec1430c0 2024-03-25T18:18:51.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.886+0000 7f27e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:51.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.888+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 937807560 0 0) 0x7f27d8004eb0 con 0x7f27ec0a98f0 2024-03-25T18:18:51.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.888+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27ec0b7c80 con 0x7f27ec0a98f0 2024-03-25T18:18:51.890 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.888+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f27dc001a30 con 0x7f27ec0a98f0 2024-03-25T18:18:51.890 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.888+0000 7f27e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:51.890 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.888+0000 7f27e8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:51.890 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.888+0000 7f27e8ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:51.890 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:51.890 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:51.890 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:51.890 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:51.890 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:51.890 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:51.890 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:51.890 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:51.890 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:51.890 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:51.890 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.888+0000 7f27e8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:51.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3002515823 0 0) 0x7f27ec0b7c80 con 0x7f27ec0a98f0 2024-03-25T18:18:51.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 >> v1:172.21.15.63:6790/0 conn(0x7f27ec1430c0 legacy=0x7f27ec14dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:51.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 >> v1:172.21.15.63:6789/0 conn(0x7f27ec13f710 legacy=0x7f27ec14d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:51.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27e8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:18:51.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27e8ff9640 1 monclient: found mon.b 2024-03-25T18:18:51.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27e8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:51.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27e8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:18:51.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec13b970 con 0x7f27ec0a98f0 2024-03-25T18:18:51.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27e8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:18:51.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27e8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:18:51.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:51.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27f274d640 5 monclient: authenticate success, global_id 4516 2024-03-25T18:18:51.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27f274d640 10 monclient: _renew_subs 2024-03-25T18:18:51.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27f274d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:18:51.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27f274d640 1 -- 172.21.15.63:0/3953020800 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ec13bc50 con 0x7f27ec0a98f0 2024-03-25T18:18:51.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27f274d640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:51.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27f274d640 10 monclient: _renew_subs 2024-03-25T18:18:51.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27f274d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:18:51.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27f274d640 1 -- 172.21.15.63:0/3953020800 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ec13a510 con 0x7f27ec0a98f0 2024-03-25T18:18:51.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.895+0000 7f27f274d640 1 librados: init done 2024-03-25T18:18:51.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.896+0000 7f27f274d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:51.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.896+0000 7f27f274d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:51.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.896+0000 7f27f274d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:18:51.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:51.896+0000 7f27f274d640 1 -- 172.21.15.63:0/3953020800 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ec0627b0 con 0x7f27ec0a98f0 2024-03-25T18:18:52.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.175+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27dc001c70 con 0x7f27ec0a98f0 2024-03-25T18:18:52.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.175+0000 7f27e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:52.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.175+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f27dc003a60 con 0x7f27ec0a98f0 2024-03-25T18:18:52.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.175+0000 7f27e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:52.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.175+0000 7f27e8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:52.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.175+0000 7f27e8ff9640 10 monclient: dump: 2024-03-25T18:18:52.177 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:52.177 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:52.177 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:52.177 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:52.177 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:52.177 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:52.178 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:52.178 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:52.178 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:52.178 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:52.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.175+0000 7f27ea7fc640 4 set_mon_vals no callback set 2024-03-25T18:18:52.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.176+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f27dc003c80 con 0x7f27ec0a98f0 2024-03-25T18:18:52.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.177+0000 7f27eb7fe640 10 monclient: get_auth_request con 0x7f27d806dd10 auth_method 0 2024-03-25T18:18:52.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.177+0000 7f27e8ff9640 10 client.4516.objecter ms_handle_connect 0x7f27d806dd10 2024-03-25T18:18:52.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.182+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f27dc054880 con 0x7f27ec0a98f0 2024-03-25T18:18:52.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.182+0000 7f27e8ff9640 10 client.4516.objecter ms_dispatch 0x7f27ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:52.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.182+0000 7f27e8ff9640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:52.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.182+0000 7f27e8ff9640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:52.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.182+0000 7f27e8ff9640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T18:18:52.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.183+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27dc050390 con 0x7f27ec0a98f0 2024-03-25T18:18:52.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.183+0000 7f27e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:52.184 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.183+0000 7f27e8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:52.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.428+0000 7f27f274d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:52.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.428+0000 7f27f274d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:52.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.428+0000 7f27f274d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:18:52.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.428+0000 7f27f274d640 1 -- 172.21.15.63:0/3953020800 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27ec13d730 con 0x7f27ec0a98f0 2024-03-25T18:18:52.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.554+0000 7f27e8ff9640 1 -- 172.21.15.63:0/3953020800 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f27dc0522c0 con 0x7f27ec0a98f0 2024-03-25T18:18:52.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.554+0000 7f27e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:52.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.554+0000 7f27e8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:52.556 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:18:52.556 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:52.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.557+0000 7f27f274d640 10 librados: watch_flush enter 2024-03-25T18:18:52.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.557+0000 7f27f274d640 10 librados: watch_flush exit 2024-03-25T18:18:52.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.557+0000 7f27f274d640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T18:18:52.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.557+0000 7f27f274d640 10 client.4516.objecter successfully canceled tick 2024-03-25T18:18:52.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.557+0000 7f27f274d640 1 -- 172.21.15.63:0/3953020800 >> v1:172.21.15.81:6800/3640340873 conn(0x7f27d806dd10 legacy=0x7f27d80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:52.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.557+0000 7f27f274d640 10 monclient: shutdown 2024-03-25T18:18:52.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.557+0000 7f27f274d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:52.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.557+0000 7f27f274d640 1 -- 172.21.15.63:0/3953020800 >> v1:172.21.15.81:6789/0 conn(0x7f27ec0a98f0 legacy=0x7f27ec13c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:52.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.557+0000 7f27f274d640 1 -- 172.21.15.63:0/3953020800 shutdown_connections 2024-03-25T18:18:52.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.557+0000 7f27f274d640 1 -- 172.21.15.63:0/3953020800 >> 172.21.15.63:0/3953020800 conn(0x7f27ec0b7ea0 msgr2=0x7f27ec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:52.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.557+0000 7f27f274d640 1 -- 172.21.15.63:0/3953020800 shutdown_connections 2024-03-25T18:18:52.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.557+0000 7f27f274d640 1 -- 172.21.15.63:0/3953020800 wait complete. 2024-03-25T18:18:52.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:52.557+0000 7f27f274d640 1 librados: shutdown 2024-03-25T18:18:52.569 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:59.207 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:40.899 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:59.217 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:40.909 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:59.254 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:40.947 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:59.264 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:06:40.924 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:59.271 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:40.963 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:59.274 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:40.967 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:59.291 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:06:40.951 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:59.313 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:06:40.974 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:59.313 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:41.005 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:59.377 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:59.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.490+0000 7fc674bb5640 10 monclient: get_monmap_and_config 2024-03-25T18:18:59.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.490+0000 7fc674bb5640 10 monclient: build_initial_monmap 2024-03-25T18:18:59.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.490+0000 7fc674bb5640 10 monclient: monmap: 2024-03-25T18:18:59.491 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:18:59.491 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:59.491 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:18:59.491255+0000 2024-03-25T18:18:59.491 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:18:59.491255+0000 2024-03-25T18:18:59.491 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:59.492 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:59.492 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:18:59.492 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:18:59.492 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:18:59.492 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:59.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.490+0000 7fc674bb5640 1 Processor -- start 2024-03-25T18:18:59.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.490+0000 7fc674bb5640 1 -- start start 2024-03-25T18:18:59.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.490+0000 7fc674bb5640 10 monclient: init 2024-03-25T18:18:59.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.491+0000 7fc674bb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:59.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.491+0000 7fc674bb5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:59.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.491+0000 7fc674bb5640 10 monclient(hunting): picked mon.noname-c con 0x7fc6700a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:18:59.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.491+0000 7fc674bb5640 10 monclient(hunting): picked mon.noname-b con 0x7fc67013f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:18:59.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.491+0000 7fc674bb5640 10 monclient(hunting): picked mon.noname-a con 0x7fc6701430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:18:59.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.491+0000 7fc674bb5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc670142520 con 0x7fc6701430c0 2024-03-25T18:18:59.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.491+0000 7fc674bb5640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6700b28a0 con 0x7fc6700a98f0 2024-03-25T18:18:59.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.491+0000 7fc674bb5640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6700b2a30 con 0x7fc67013f710 2024-03-25T18:18:59.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.491+0000 7fc674bb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:59.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.491+0000 7fc674bb5640 10 monclient(hunting): authenticate will time out at 7833.678151s 2024-03-25T18:18:59.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.503+0000 7fc66e59c640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc6700a98f0 0x7fc6700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:59742/0 (socket says 172.21.15.63:59742) 2024-03-25T18:18:59.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.503+0000 7fc66e59c640 1 -- 172.21.15.63:0/594650782 learned_addr learned my addr 172.21.15.63:0/594650782 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:18:59.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.510+0000 7fc66d59a640 1 -- 172.21.15.63:0/594650782 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474753719 0 0) 0x7fc6700b28a0 con 0x7fc6700a98f0 2024-03-25T18:18:59.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.510+0000 7fc66d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:59.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.510+0000 7fc66d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:59.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.510+0000 7fc66d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:59.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.510+0000 7fc66d59a640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:18:59.512 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.510+0000 7fc66d59a640 1 -- 172.21.15.63:0/594650782 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc658003520 con 0x7fc6700a98f0 2024-03-25T18:18:59.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc66d59a640 1 -- 172.21.15.63:0/594650782 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2769616027 0 0) 0x7fc658003520 con 0x7fc6700a98f0 2024-03-25T18:18:59.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc66d59a640 1 -- 172.21.15.63:0/594650782 >> v1:172.21.15.63:6790/0 conn(0x7fc67013f710 legacy=0x7fc6701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:18:59.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc66d59a640 1 -- 172.21.15.63:0/594650782 >> v1:172.21.15.63:6789/0 conn(0x7fc6701430c0 legacy=0x7fc67014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:59.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc66d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:18:59.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc66d59a640 1 monclient: found mon.noname-c 2024-03-25T18:18:59.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc66d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:59.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc66d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:18:59.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc66d59a640 1 -- 172.21.15.63:0/594650782 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6700b44c0 con 0x7fc6700a98f0 2024-03-25T18:18:59.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc66d59a640 10 monclient: _finish_auth 0 2024-03-25T18:18:59.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc66d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:18:59.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc66d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:59.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc66d59a640 1 -- 172.21.15.63:0/594650782 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc660001690 con 0x7fc6700a98f0 2024-03-25T18:18:59.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc66d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:59.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc66d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:59.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc66d59a640 10 monclient: dump: 2024-03-25T18:18:59.518 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:59.518 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:59.518 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:59.518 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:59.518 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:59.518 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:59.519 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:59.519 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:59.519 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:59.519 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:59.519 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc674bb5640 5 monclient: authenticate success, global_id 4519 2024-03-25T18:18:59.519 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.515+0000 7fc674bb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:59.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.524+0000 7fc66d59a640 1 -- 172.21.15.63:0/594650782 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc660001860 con 0x7fc6700a98f0 2024-03-25T18:18:59.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.524+0000 7fc66d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:59.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.524+0000 7fc66d59a640 1 -- 172.21.15.63:0/594650782 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc660001cb0 con 0x7fc6700a98f0 2024-03-25T18:18:59.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.524+0000 7fc66d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:59.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.524+0000 7fc66d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:59.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.524+0000 7fc66d59a640 10 monclient: dump: 2024-03-25T18:18:59.526 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:59.526 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:59.526 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:59.526 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:59.526 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:59.527 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:59.527 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:59.527 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:59.527 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:59.527 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:59.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.524+0000 7fc674bb5640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:59.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.524+0000 7fc674bb5640 4 set_mon_vals no callback set 2024-03-25T18:18:59.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.525+0000 7fc674bb5640 10 monclient: shutdown 2024-03-25T18:18:59.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.525+0000 7fc674bb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:59.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.525+0000 7fc674bb5640 1 -- 172.21.15.63:0/594650782 >> v1:172.21.15.81:6789/0 conn(0x7fc6700a98f0 legacy=0x7fc6700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:59.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.525+0000 7fc674bb5640 1 -- 172.21.15.63:0/594650782 shutdown_connections 2024-03-25T18:18:59.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.525+0000 7fc674bb5640 1 -- 172.21.15.63:0/594650782 >> 172.21.15.63:0/594650782 conn(0x7fc6700b7ea0 msgr2=0x7fc6700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:59.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.525+0000 7fc674bb5640 1 -- 172.21.15.63:0/594650782 shutdown_connections 2024-03-25T18:18:59.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.525+0000 7fc674bb5640 1 -- 172.21.15.63:0/594650782 wait complete. 2024-03-25T18:18:59.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.526+0000 7fc674bb5640 10 monclient: build_initial_monmap 2024-03-25T18:18:59.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.526+0000 7fc674bb5640 10 monclient: monmap: 2024-03-25T18:18:59.530 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:18:59.530 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:59.530 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:18:59.530 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:18:59.531 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:59.531 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:18:59.531 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:18:59.531 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:18:59.531 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:18:59.531 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:59.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.527+0000 7fc674bb5640 1 librados: starting msgr at 2024-03-25T18:18:59.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.527+0000 7fc674bb5640 1 librados: starting objecter 2024-03-25T18:18:59.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.527+0000 7fc674bb5640 1 Processor -- start 2024-03-25T18:18:59.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.527+0000 7fc674bb5640 1 -- start start 2024-03-25T18:18:59.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.527+0000 7fc674bb5640 1 librados: setting wanted keys 2024-03-25T18:18:59.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.527+0000 7fc674bb5640 1 librados: calling monclient init 2024-03-25T18:18:59.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.527+0000 7fc674bb5640 10 monclient: init 2024-03-25T18:18:59.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.528+0000 7fc674bb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:59.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.528+0000 7fc674bb5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:59.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.528+0000 7fc674bb5640 10 monclient(hunting): picked mon.noname-c con 0x7fc6700a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:18:59.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.528+0000 7fc674bb5640 10 monclient(hunting): picked mon.noname-b con 0x7fc67013f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:18:59.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.528+0000 7fc674bb5640 10 monclient(hunting): picked mon.noname-a con 0x7fc6701430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:18:59.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.528+0000 7fc674bb5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc670139840 con 0x7fc6701430c0 2024-03-25T18:18:59.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.528+0000 7fc674bb5640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc670138b90 con 0x7fc67013f710 2024-03-25T18:18:59.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.528+0000 7fc674bb5640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc670138790 con 0x7fc6700a98f0 2024-03-25T18:18:59.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.528+0000 7fc674bb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:59.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.528+0000 7fc674bb5640 10 monclient(hunting): authenticate will time out at 7833.714748s 2024-03-25T18:18:59.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.539+0000 7fc66dd9b640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc67013f710 0x7fc6701d7790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:59766/0 (socket says 172.21.15.63:59766) 2024-03-25T18:18:59.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.539+0000 7fc66dd9b640 1 -- 172.21.15.63:0/2465663323 learned_addr learned my addr 172.21.15.63:0/2465663323 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:18:59.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.540+0000 7fc65effd640 10 client.?.objecter ms_handle_connect 0x7fc67013f710 2024-03-25T18:18:59.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.540+0000 7fc65effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:59.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.542+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279268195 0 0) 0x7fc670138b90 con 0x7fc67013f710 2024-03-25T18:18:59.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.542+0000 7fc65effd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:59.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.542+0000 7fc65effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:59.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.542+0000 7fc65effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:59.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.542+0000 7fc65effd640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:18:59.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.542+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc650004eb0 con 0x7fc67013f710 2024-03-25T18:18:59.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.553+0000 7fc65effd640 10 client.?.objecter ms_handle_connect 0x7fc6701430c0 2024-03-25T18:18:59.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.553+0000 7fc65effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:59.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.553+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1053455476 0 0) 0x7fc650004eb0 con 0x7fc67013f710 2024-03-25T18:18:59.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.553+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc670138b90 con 0x7fc67013f710 2024-03-25T18:18:59.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.553+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc664001960 con 0x7fc67013f710 2024-03-25T18:18:59.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.553+0000 7fc65effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:59.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.553+0000 7fc65effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:59.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.553+0000 7fc65effd640 10 monclient(hunting): dump: 2024-03-25T18:18:59.555 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:59.555 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:59.555 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:59.556 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:59.556 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:59.556 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:59.556 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:59.556 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:59.556 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:59.556 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:59.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.553+0000 7fc65effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:59.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.555+0000 7fc65effd640 10 client.?.objecter ms_handle_connect 0x7fc6700a98f0 2024-03-25T18:18:59.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.555+0000 7fc65effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:59.564 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.563+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928514720 0 0) 0x7fc670139840 con 0x7fc6701430c0 2024-03-25T18:18:59.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.563+0000 7fc65effd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:59.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.563+0000 7fc65effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:59.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.563+0000 7fc65effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:59.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.563+0000 7fc65effd640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:18:59.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.563+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc650004eb0 con 0x7fc6701430c0 2024-03-25T18:18:59.571 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 291021369 0 0) 0x7fc670138b90 con 0x7fc67013f710 2024-03-25T18:18:59.571 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 >> v1:172.21.15.63:6790/0 conn(0x7fc6700a98f0 legacy=0x7fc670138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:59.571 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 >> v1:172.21.15.63:6789/0 conn(0x7fc6701430c0 legacy=0x7fc6701d7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:59.571 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc65effd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:59.571 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc65effd640 1 monclient: found mon.b 2024-03-25T18:18:59.571 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc65effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:59.571 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc65effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:18:59.571 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc670138920 con 0x7fc67013f710 2024-03-25T18:18:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc65effd640 10 monclient: _finish_auth 0 2024-03-25T18:18:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc65effd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc65effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc674bb5640 5 monclient: authenticate success, global_id 4522 2024-03-25T18:18:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc674bb5640 10 monclient: _renew_subs 2024-03-25T18:18:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc674bb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:18:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc674bb5640 1 -- 172.21.15.63:0/2465663323 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6700b15a0 con 0x7fc67013f710 2024-03-25T18:18:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc674bb5640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc674bb5640 10 monclient: _renew_subs 2024-03-25T18:18:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc674bb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:18:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc674bb5640 1 -- 172.21.15.63:0/2465663323 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6701e3230 con 0x7fc67013f710 2024-03-25T18:18:59.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.569+0000 7fc674bb5640 1 librados: init done 2024-03-25T18:18:59.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.570+0000 7fc674bb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:59.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.570+0000 7fc674bb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:59.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.570+0000 7fc674bb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:18:59.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.570+0000 7fc674bb5640 1 -- 172.21.15.63:0/2465663323 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6700627b0 con 0x7fc67013f710 2024-03-25T18:18:59.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.835+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc664001c40 con 0x7fc67013f710 2024-03-25T18:18:59.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.835+0000 7fc65effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:59.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.835+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc664003a40 con 0x7fc67013f710 2024-03-25T18:18:59.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.835+0000 7fc65effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:59.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.835+0000 7fc65effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:59.837 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.835+0000 7fc65effd640 10 monclient: dump: 2024-03-25T18:18:59.837 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:18:59.837 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:18:59.837 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:59.837 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:18:59.837 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:59.837 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:18:59.837 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:18:59.837 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:18:59.837 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:18:59.838 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:18:59.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.835+0000 7fc66cd99640 4 set_mon_vals no callback set 2024-03-25T18:18:59.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.836+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fc664003c60 con 0x7fc67013f710 2024-03-25T18:18:59.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.836+0000 7fc66e59c640 10 monclient: get_auth_request con 0x7fc65006de20 auth_method 0 2024-03-25T18:18:59.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.837+0000 7fc65effd640 10 client.4522.objecter ms_handle_connect 0x7fc65006de20 2024-03-25T18:18:59.842 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.840+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fc664054700 con 0x7fc67013f710 2024-03-25T18:18:59.842 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.840+0000 7fc65effd640 10 client.4522.objecter ms_dispatch 0x7fc670000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:59.842 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.840+0000 7fc65effd640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:59.842 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.840+0000 7fc65effd640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:59.842 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.841+0000 7fc65effd640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-25T18:18:59.842 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.841+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc664050210 con 0x7fc67013f710 2024-03-25T18:18:59.842 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.841+0000 7fc65effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:59.843 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:18:59.841+0000 7fc65effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:00.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.097+0000 7fc674bb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:00.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.097+0000 7fc674bb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:00.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.097+0000 7fc674bb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:00.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.097+0000 7fc674bb5640 1 -- 172.21.15.63:0/2465663323 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc670138d20 con 0x7fc67013f710 2024-03-25T18:19:00.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.130+0000 7fc65effd640 1 -- 172.21.15.63:0/2465663323 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fc664052140 con 0x7fc67013f710 2024-03-25T18:19:00.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.130+0000 7fc65effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:00.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.130+0000 7fc65effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:00.132 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:19:00.132 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:00.134 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.132+0000 7fc674bb5640 10 librados: watch_flush enter 2024-03-25T18:19:00.134 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.132+0000 7fc674bb5640 10 librados: watch_flush exit 2024-03-25T18:19:00.134 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.132+0000 7fc674bb5640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-25T18:19:00.134 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.132+0000 7fc674bb5640 10 client.4522.objecter successfully canceled tick 2024-03-25T18:19:00.134 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.133+0000 7fc674bb5640 1 -- 172.21.15.63:0/2465663323 >> v1:172.21.15.81:6800/3640340873 conn(0x7fc65006de20 legacy=0x7fc6500702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.133+0000 7fc674bb5640 10 monclient: shutdown 2024-03-25T18:19:00.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.133+0000 7fc674bb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:00.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.133+0000 7fc674bb5640 1 -- 172.21.15.63:0/2465663323 >> v1:172.21.15.81:6789/0 conn(0x7fc67013f710 legacy=0x7fc6701d7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.133+0000 7fc674bb5640 1 -- 172.21.15.63:0/2465663323 shutdown_connections 2024-03-25T18:19:00.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.133+0000 7fc674bb5640 1 -- 172.21.15.63:0/2465663323 >> 172.21.15.63:0/2465663323 conn(0x7fc6700b7ea0 msgr2=0x7fc670074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:00.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.133+0000 7fc674bb5640 1 -- 172.21.15.63:0/2465663323 shutdown_connections 2024-03-25T18:19:00.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.133+0000 7fc674bb5640 1 -- 172.21.15.63:0/2465663323 wait complete. 2024-03-25T18:19:00.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:00.133+0000 7fc674bb5640 1 librados: shutdown 2024-03-25T18:19:00.145 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:06.952 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:07.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.068+0000 7f16019af640 10 monclient: get_monmap_and_config 2024-03-25T18:19:07.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.068+0000 7f16019af640 10 monclient: build_initial_monmap 2024-03-25T18:19:07.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.068+0000 7f16019af640 10 monclient: monmap: 2024-03-25T18:19:07.069 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:19:07.069 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:07.069 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:19:07.069241+0000 2024-03-25T18:19:07.069 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:19:07.069241+0000 2024-03-25T18:19:07.070 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:07.070 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:07.070 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:19:07.070 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:19:07.070 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:19:07.070 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:07.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.068+0000 7f16019af640 1 Processor -- start 2024-03-25T18:19:07.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.068+0000 7f16019af640 1 -- start start 2024-03-25T18:19:07.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.068+0000 7f16019af640 10 monclient: init 2024-03-25T18:19:07.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.069+0000 7f16019af640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:07.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.069+0000 7f16019af640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:07.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.069+0000 7f16019af640 10 monclient(hunting): picked mon.noname-a con 0x7f15fc0a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:19:07.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.069+0000 7f16019af640 10 monclient(hunting): picked mon.noname-b con 0x7f15fc13f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:19:07.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.069+0000 7f16019af640 10 monclient(hunting): picked mon.noname-c con 0x7f15fc1430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:19:07.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.069+0000 7f16019af640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15fc142520 con 0x7f15fc0a98f0 2024-03-25T18:19:07.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.069+0000 7f16019af640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15fc0b28a0 con 0x7f15fc1430c0 2024-03-25T18:19:07.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.069+0000 7f16019af640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15fc0b2a30 con 0x7f15fc13f710 2024-03-25T18:19:07.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.069+0000 7f16019af640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:07.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.069+0000 7f16019af640 10 monclient(hunting): authenticate will time out at 7841.256039s 2024-03-25T18:19:07.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.083+0000 7f15faffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f15fc0a98f0 0x7f15fc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:43788/0 (socket says 172.21.15.63:43788) 2024-03-25T18:19:07.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.083+0000 7f15faffd640 1 -- 172.21.15.63:0/3625680661 learned_addr learned my addr 172.21.15.63:0/3625680661 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:19:07.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.103+0000 7f15f9ffb640 1 -- 172.21.15.63:0/3625680661 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421965433 0 0) 0x7f15fc142520 con 0x7f15fc0a98f0 2024-03-25T18:19:07.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.103+0000 7f15f9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:07.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.103+0000 7f15f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:07.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.103+0000 7f15f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:07.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.103+0000 7f15f9ffb640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:19:07.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.103+0000 7f15f9ffb640 1 -- 172.21.15.63:0/3625680661 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e4003520 con 0x7f15fc0a98f0 2024-03-25T18:19:07.106 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.103+0000 7f15f9ffb640 1 -- 172.21.15.63:0/3625680661 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940739612 0 0) 0x7f15fc0b2a30 con 0x7f15fc13f710 2024-03-25T18:19:07.106 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.103+0000 7f15f9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:07.106 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.103+0000 7f15f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:07.106 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.103+0000 7f15f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:07.106 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.103+0000 7f15f9ffb640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:19:07.106 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.103+0000 7f15f9ffb640 1 -- 172.21.15.63:0/3625680661 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15fc142520 con 0x7f15fc13f710 2024-03-25T18:19:07.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.112+0000 7f15f9ffb640 1 -- 172.21.15.63:0/3625680661 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2298529825 0 0) 0x7f15e4003520 con 0x7f15fc0a98f0 2024-03-25T18:19:07.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.112+0000 7f15f9ffb640 1 -- 172.21.15.63:0/3625680661 >> v1:172.21.15.63:6790/0 conn(0x7f15fc13f710 legacy=0x7f15fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:07.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.112+0000 7f15f9ffb640 1 -- 172.21.15.63:0/3625680661 >> v1:172.21.15.81:6789/0 conn(0x7f15fc1430c0 legacy=0x7f15fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:07.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.112+0000 7f15f9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:07.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.112+0000 7f15f9ffb640 1 monclient: found mon.noname-a 2024-03-25T18:19:07.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.112+0000 7f15f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:07.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.112+0000 7f15f9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:19:07.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.112+0000 7f15f9ffb640 1 -- 172.21.15.63:0/3625680661 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15fc0b44c0 con 0x7f15fc0a98f0 2024-03-25T18:19:07.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.112+0000 7f15f9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:07.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.112+0000 7f15f9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:07.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.112+0000 7f15f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:07.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.112+0000 7f16019af640 5 monclient: authenticate success, global_id 4497 2024-03-25T18:19:07.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.112+0000 7f16019af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:07.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.130+0000 7f15f9ffb640 1 -- 172.21.15.63:0/3625680661 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f15ec001650 con 0x7f15fc0a98f0 2024-03-25T18:19:07.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.130+0000 7f15f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:07.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.130+0000 7f15f9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:07.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.130+0000 7f15f9ffb640 10 monclient: dump: 2024-03-25T18:19:07.132 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:07.132 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:07.132 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:07.132 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:07.132 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:07.132 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:07.132 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:07.132 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:07.132 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:07.132 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:07.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.130+0000 7f15f9ffb640 1 -- 172.21.15.63:0/3625680661 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15ec0018a0 con 0x7f15fc0a98f0 2024-03-25T18:19:07.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.130+0000 7f16019af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:07.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.130+0000 7f15f9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:07.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.130+0000 7f16019af640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:07.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.130+0000 7f15f9ffb640 1 -- 172.21.15.63:0/3625680661 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f15ec001c30 con 0x7f15fc0a98f0 2024-03-25T18:19:07.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.130+0000 7f15f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:07.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.130+0000 7f16019af640 4 set_mon_vals no callback set 2024-03-25T18:19:07.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.130+0000 7f15f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:07.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.130+0000 7f15f9ffb640 10 monclient: dump: 2024-03-25T18:19:07.133 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:07.133 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:07.133 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:07.133 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:07.133 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:07.133 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:07.133 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:07.133 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:07.133 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:07.134 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:07.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.131+0000 7f16019af640 10 monclient: shutdown 2024-03-25T18:19:07.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.131+0000 7f16019af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:07.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.131+0000 7f16019af640 1 -- 172.21.15.63:0/3625680661 >> v1:172.21.15.63:6789/0 conn(0x7f15fc0a98f0 legacy=0x7f15fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:07.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.131+0000 7f16019af640 1 -- 172.21.15.63:0/3625680661 shutdown_connections 2024-03-25T18:19:07.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.131+0000 7f16019af640 1 -- 172.21.15.63:0/3625680661 >> 172.21.15.63:0/3625680661 conn(0x7f15fc0b7ea0 msgr2=0x7f15fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:07.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.131+0000 7f16019af640 1 -- 172.21.15.63:0/3625680661 shutdown_connections 2024-03-25T18:19:07.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.131+0000 7f16019af640 1 -- 172.21.15.63:0/3625680661 wait complete. 2024-03-25T18:19:07.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.132+0000 7f16019af640 10 monclient: build_initial_monmap 2024-03-25T18:19:07.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.132+0000 7f16019af640 10 monclient: monmap: 2024-03-25T18:19:07.136 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:19:07.136 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:07.136 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:19:07.136 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:19:07.136 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:07.137 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:19:07.137 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:19:07.137 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:19:07.137 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:19:07.137 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:07.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.132+0000 7f16019af640 1 librados: starting msgr at 2024-03-25T18:19:07.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.132+0000 7f16019af640 1 librados: starting objecter 2024-03-25T18:19:07.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.133+0000 7f16019af640 1 Processor -- start 2024-03-25T18:19:07.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.133+0000 7f16019af640 1 -- start start 2024-03-25T18:19:07.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.133+0000 7f16019af640 1 librados: setting wanted keys 2024-03-25T18:19:07.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.133+0000 7f16019af640 1 librados: calling monclient init 2024-03-25T18:19:07.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.133+0000 7f16019af640 10 monclient: init 2024-03-25T18:19:07.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.133+0000 7f16019af640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:07.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.133+0000 7f16019af640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:07.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.133+0000 7f16019af640 10 monclient(hunting): picked mon.noname-a con 0x7f15fc0a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:19:07.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.133+0000 7f16019af640 10 monclient(hunting): picked mon.noname-b con 0x7f15fc13f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:19:07.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.133+0000 7f16019af640 10 monclient(hunting): picked mon.noname-c con 0x7f15fc1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:19:07.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.133+0000 7f16019af640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15fc136d30 con 0x7f15fc0a98f0 2024-03-25T18:19:07.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.133+0000 7f16019af640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15fc137010 con 0x7f15fc13f710 2024-03-25T18:19:07.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.133+0000 7f16019af640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15fc1372f0 con 0x7f15fc1430c0 2024-03-25T18:19:07.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.133+0000 7f16019af640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:07.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.133+0000 7f16019af640 10 monclient(hunting): authenticate will time out at 7841.320245s 2024-03-25T18:19:07.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.143+0000 7f15faffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f15fc0a98f0 0x7f15fc1379f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:43834/0 (socket says 172.21.15.63:43834) 2024-03-25T18:19:07.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.143+0000 7f15faffd640 1 -- 172.21.15.63:0/2235436657 learned_addr learned my addr 172.21.15.63:0/2235436657 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:19:07.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.146+0000 7f15e3fff640 10 client.?.objecter ms_handle_connect 0x7f15fc0a98f0 2024-03-25T18:19:07.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.146+0000 7f15e3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:07.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.156+0000 7f15e3fff640 10 client.?.objecter ms_handle_connect 0x7f15fc13f710 2024-03-25T18:19:07.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.156+0000 7f15e3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:07.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.162+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506318654 0 0) 0x7f15fc136d30 con 0x7f15fc0a98f0 2024-03-25T18:19:07.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.162+0000 7f15e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:07.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.162+0000 7f15e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:07.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.162+0000 7f15e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:07.164 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.162+0000 7f15e3fff640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:19:07.164 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.162+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e8004eb0 con 0x7f15fc0a98f0 2024-03-25T18:19:07.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.165+0000 7f15e3fff640 10 client.?.objecter ms_handle_connect 0x7f15fc1430c0 2024-03-25T18:19:07.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.165+0000 7f15e3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:07.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.173+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233499872 0 0) 0x7f15fc137010 con 0x7f15fc13f710 2024-03-25T18:19:07.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.173+0000 7f15e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:07.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.173+0000 7f15e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:07.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.173+0000 7f15e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:07.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.173+0000 7f15e3fff640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:19:07.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.173+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15fc136d30 con 0x7f15fc13f710 2024-03-25T18:19:07.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.174+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459040641 0 0) 0x7f15fc1372f0 con 0x7f15fc1430c0 2024-03-25T18:19:07.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.174+0000 7f15e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:07.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.174+0000 7f15e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:07.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.174+0000 7f15e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:07.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.174+0000 7f15e3fff640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:19:07.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.174+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15fc137010 con 0x7f15fc1430c0 2024-03-25T18:19:07.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.175+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1157977439 0 0) 0x7f15e8004eb0 con 0x7f15fc0a98f0 2024-03-25T18:19:07.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.175+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15fc1372f0 con 0x7f15fc0a98f0 2024-03-25T18:19:07.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.175+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f15ec001bb0 con 0x7f15fc0a98f0 2024-03-25T18:19:07.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.175+0000 7f15e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:07.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.175+0000 7f15e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:07.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.175+0000 7f15e3fff640 10 monclient(hunting): dump: 2024-03-25T18:19:07.177 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:07.177 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:07.177 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:07.177 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:07.177 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:07.177 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:07.177 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:07.177 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:07.177 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:07.178 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:07.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.175+0000 7f15e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:07.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.195+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 882146572 0 0) 0x7f15fc137010 con 0x7f15fc1430c0 2024-03-25T18:19:07.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.195+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15e8004eb0 con 0x7f15fc1430c0 2024-03-25T18:19:07.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.195+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f15f4001ad0 con 0x7f15fc1430c0 2024-03-25T18:19:07.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.195+0000 7f15e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:07.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.195+0000 7f15e3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:07.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.195+0000 7f15e3fff640 10 monclient(hunting): dump: 2024-03-25T18:19:07.196 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:07.196 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:07.196 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:07.196 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:07.196 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:07.196 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:07.197 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:07.197 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:07.197 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:07.197 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:07.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.200+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 326045386 0 0) 0x7f15fc136d30 con 0x7f15fc13f710 2024-03-25T18:19:07.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.200+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15fc137010 con 0x7f15fc13f710 2024-03-25T18:19:07.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.200+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f15f0001930 con 0x7f15fc13f710 2024-03-25T18:19:07.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.200+0000 7f15e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:07.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.200+0000 7f15e3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:07.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.200+0000 7f15e3fff640 10 monclient(hunting): dump: 2024-03-25T18:19:07.201 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:07.201 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:07.201 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:07.201 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:07.201 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:07.201 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:07.202 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:07.202 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:07.202 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:07.202 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:07.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2871500088 0 0) 0x7f15fc1372f0 con 0x7f15fc0a98f0 2024-03-25T18:19:07.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 >> v1:172.21.15.63:6790/0 conn(0x7f15fc1430c0 legacy=0x7f15fc0b6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:07.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 >> v1:172.21.15.81:6789/0 conn(0x7f15fc13f710 legacy=0x7f15fc135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:07.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f15e3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:07.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f15e3fff640 1 monclient: found mon.a 2024-03-25T18:19:07.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f15e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:07.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f15e3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:19:07.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15fc1375d0 con 0x7f15fc0a98f0 2024-03-25T18:19:07.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f15e3fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:07.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f15e3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:07.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f15e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:07.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f16019af640 5 monclient: authenticate success, global_id 4500 2024-03-25T18:19:07.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f16019af640 10 monclient: _renew_subs 2024-03-25T18:19:07.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f16019af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:19:07.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f16019af640 1 -- 172.21.15.63:0/2235436657 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15fc135c00 con 0x7f15fc0a98f0 2024-03-25T18:19:07.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f16019af640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:07.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f16019af640 10 monclient: _renew_subs 2024-03-25T18:19:07.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f16019af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:19:07.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f16019af640 1 -- 172.21.15.63:0/2235436657 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15fc1e3690 con 0x7f15fc0a98f0 2024-03-25T18:19:07.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.203+0000 7f16019af640 1 librados: init done 2024-03-25T18:19:07.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.205+0000 7f16019af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:07.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.205+0000 7f16019af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:07.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.205+0000 7f16019af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:19:07.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.205+0000 7f16019af640 1 -- 172.21.15.63:0/2235436657 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15fc0627b0 con 0x7f15fc0a98f0 2024-03-25T18:19:07.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.504+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15ec004bf0 con 0x7f15fc0a98f0 2024-03-25T18:19:07.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.504+0000 7f15e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:07.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.504+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f15ec004f00 con 0x7f15fc0a98f0 2024-03-25T18:19:07.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.504+0000 7f15e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:07.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.504+0000 7f15e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:07.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.504+0000 7f15e3fff640 10 monclient: dump: 2024-03-25T18:19:07.506 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:07.506 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:07.506 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:07.506 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:07.506 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:07.506 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:07.506 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:07.506 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:07.506 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:07.506 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:07.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.504+0000 7f15f97fa640 4 set_mon_vals no callback set 2024-03-25T18:19:07.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.505+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f15ec01aed0 con 0x7f15fc0a98f0 2024-03-25T18:19:07.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.505+0000 7f15fa7fc640 10 monclient: get_auth_request con 0x7f15e806e0a0 auth_method 0 2024-03-25T18:19:07.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.506+0000 7f15e3fff640 10 client.4500.objecter ms_handle_connect 0x7f15e806e0a0 2024-03-25T18:19:07.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.512+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f15ec054950 con 0x7f15fc0a98f0 2024-03-25T18:19:07.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.512+0000 7f15e3fff640 10 client.4500.objecter ms_dispatch 0x7f15fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:07.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.512+0000 7f15e3fff640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:07.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.512+0000 7f15e3fff640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:07.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.513+0000 7f15e3fff640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-25T18:19:07.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.513+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15ec050460 con 0x7f15fc0a98f0 2024-03-25T18:19:07.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.513+0000 7f15e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:07.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.513+0000 7f15e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:07.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.764+0000 7f16019af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:07.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.764+0000 7f16019af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:07.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.764+0000 7f16019af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:19:07.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.764+0000 7f16019af640 1 -- 172.21.15.63:0/2235436657 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15fc13df50 con 0x7f15fc0a98f0 2024-03-25T18:19:07.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.793+0000 7f15e3fff640 1 -- 172.21.15.63:0/2235436657 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f15ec052390 con 0x7f15fc0a98f0 2024-03-25T18:19:07.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.793+0000 7f15e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:07.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.793+0000 7f15e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:07.795 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:19:07.795 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:07.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.795+0000 7f16019af640 10 librados: watch_flush enter 2024-03-25T18:19:07.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.795+0000 7f16019af640 10 librados: watch_flush exit 2024-03-25T18:19:07.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.795+0000 7f16019af640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-25T18:19:07.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.795+0000 7f16019af640 10 client.4500.objecter successfully canceled tick 2024-03-25T18:19:07.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.795+0000 7f16019af640 1 -- 172.21.15.63:0/2235436657 >> v1:172.21.15.81:6800/3640340873 conn(0x7f15e806e0a0 legacy=0x7f15e8070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:07.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.795+0000 7f16019af640 10 monclient: shutdown 2024-03-25T18:19:07.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.795+0000 7f16019af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:07.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.795+0000 7f16019af640 1 -- 172.21.15.63:0/2235436657 >> v1:172.21.15.63:6789/0 conn(0x7f15fc0a98f0 legacy=0x7f15fc1379f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:07.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.795+0000 7f16019af640 1 -- 172.21.15.63:0/2235436657 shutdown_connections 2024-03-25T18:19:07.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.795+0000 7f16019af640 1 -- 172.21.15.63:0/2235436657 >> 172.21.15.63:0/2235436657 conn(0x7f15fc0b7ea0 msgr2=0x7f15fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:07.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.796+0000 7f16019af640 1 -- 172.21.15.63:0/2235436657 shutdown_connections 2024-03-25T18:19:07.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.796+0000 7f16019af640 1 -- 172.21.15.63:0/2235436657 wait complete. 2024-03-25T18:19:07.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:07.796+0000 7f16019af640 1 librados: shutdown 2024-03-25T18:19:07.808 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:14.614 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:14.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.730+0000 7f1455e5c640 10 monclient: get_monmap_and_config 2024-03-25T18:19:14.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.730+0000 7f1455e5c640 10 monclient: build_initial_monmap 2024-03-25T18:19:14.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.730+0000 7f1455e5c640 10 monclient: monmap: 2024-03-25T18:19:14.731 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:19:14.731 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:14.732 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:19:14.731381+0000 2024-03-25T18:19:14.732 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:19:14.731381+0000 2024-03-25T18:19:14.732 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:14.732 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:14.732 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:19:14.732 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:19:14.732 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:19:14.732 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:14.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.731+0000 7f1455e5c640 1 Processor -- start 2024-03-25T18:19:14.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.731+0000 7f1455e5c640 1 -- start start 2024-03-25T18:19:14.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.731+0000 7f1455e5c640 10 monclient: init 2024-03-25T18:19:14.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.732+0000 7f1455e5c640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:14.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.732+0000 7f1455e5c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:14.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.732+0000 7f1455e5c640 10 monclient(hunting): picked mon.noname-b con 0x7f14500a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:19:14.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.732+0000 7f1455e5c640 10 monclient(hunting): picked mon.noname-c con 0x7f145013f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:19:14.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.732+0000 7f1455e5c640 10 monclient(hunting): picked mon.noname-a con 0x7f14501430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:19:14.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.732+0000 7f1455e5c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1450142520 con 0x7f14501430c0 2024-03-25T18:19:14.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.732+0000 7f1455e5c640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14500b28a0 con 0x7f145013f710 2024-03-25T18:19:14.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.732+0000 7f1455e5c640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14500b2a30 con 0x7f14500a98f0 2024-03-25T18:19:14.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.732+0000 7f1455e5c640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:14.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.732+0000 7f1455e5c640 10 monclient(hunting): authenticate will time out at 7848.918509s 2024-03-25T18:19:14.743 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.742+0000 7f144ffff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f14501430c0 0x7f145014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:60746/0 (socket says 172.21.15.63:60746) 2024-03-25T18:19:14.743 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.742+0000 7f144ffff640 1 -- 172.21.15.63:0/1726997041 learned_addr learned my addr 172.21.15.63:0/1726997041 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:19:14.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.761+0000 7f144e7fc640 1 -- 172.21.15.63:0/1726997041 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062491588 0 0) 0x7f14500b28a0 con 0x7f145013f710 2024-03-25T18:19:14.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.761+0000 7f144e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:14.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.761+0000 7f144e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:14.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.761+0000 7f144e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:14.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.761+0000 7f144e7fc640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:19:14.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.761+0000 7f144e7fc640 1 -- 172.21.15.63:0/1726997041 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1438003520 con 0x7f145013f710 2024-03-25T18:19:14.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.764+0000 7f144e7fc640 1 -- 172.21.15.63:0/1726997041 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334445944 0 0) 0x7f1450142520 con 0x7f14501430c0 2024-03-25T18:19:14.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.764+0000 7f144e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:14.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.764+0000 7f144e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:14.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.764+0000 7f144e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:14.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.764+0000 7f144e7fc640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:19:14.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.764+0000 7f144e7fc640 1 -- 172.21.15.63:0/1726997041 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14500b28a0 con 0x7f14501430c0 2024-03-25T18:19:14.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.765+0000 7f144e7fc640 1 -- 172.21.15.63:0/1726997041 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833193971 0 0) 0x7f14500b2a30 con 0x7f14500a98f0 2024-03-25T18:19:14.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.765+0000 7f144e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:14.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.765+0000 7f144e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:14.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.765+0000 7f144e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:14.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.765+0000 7f144e7fc640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:19:14.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.765+0000 7f144e7fc640 1 -- 172.21.15.63:0/1726997041 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1450142520 con 0x7f14500a98f0 2024-03-25T18:19:14.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f144e7fc640 1 -- 172.21.15.63:0/1726997041 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 716530198 0 0) 0x7f14500b28a0 con 0x7f14501430c0 2024-03-25T18:19:14.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f144e7fc640 1 -- 172.21.15.63:0/1726997041 >> v1:172.21.15.63:6790/0 conn(0x7f14500a98f0 legacy=0x7f14500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:14.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f144e7fc640 1 -- 172.21.15.63:0/1726997041 >> v1:172.21.15.81:6789/0 conn(0x7f145013f710 legacy=0x7f14501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:14.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f144e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:14.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f144e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:19:14.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f144e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:14.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f144e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:19:14.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f144e7fc640 1 -- 172.21.15.63:0/1726997041 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14500b44c0 con 0x7f14501430c0 2024-03-25T18:19:14.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f144e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:14.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f144e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:14.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f144e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:14.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f144e7fc640 1 -- 172.21.15.63:0/1726997041 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f14480019f0 con 0x7f14501430c0 2024-03-25T18:19:14.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f1455e5c640 5 monclient: authenticate success, global_id 4503 2024-03-25T18:19:14.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f144e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:14.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f144e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:14.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f144e7fc640 10 monclient: dump: 2024-03-25T18:19:14.773 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:14.774 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:14.774 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:14.774 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:14.774 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:14.774 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:14.774 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:14.774 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:14.774 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:14.774 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:14.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.771+0000 7f1455e5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:14.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.788+0000 7f144e7fc640 1 -- 172.21.15.63:0/1726997041 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1448001b50 con 0x7f14501430c0 2024-03-25T18:19:14.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.788+0000 7f144e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:14.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.788+0000 7f144e7fc640 1 -- 172.21.15.63:0/1726997041 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f1448001f60 con 0x7f14501430c0 2024-03-25T18:19:14.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.788+0000 7f1455e5c640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:14.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.788+0000 7f1455e5c640 4 set_mon_vals no callback set 2024-03-25T18:19:14.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.788+0000 7f144e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:14.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.788+0000 7f144e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:14.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.788+0000 7f144e7fc640 10 monclient: dump: 2024-03-25T18:19:14.789 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:14.789 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:14.790 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:14.790 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:14.790 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:14.790 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:14.790 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:14.790 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:14.790 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:14.790 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:14.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.788+0000 7f1455e5c640 10 monclient: shutdown 2024-03-25T18:19:14.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.788+0000 7f1455e5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:14.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.788+0000 7f1455e5c640 1 -- 172.21.15.63:0/1726997041 >> v1:172.21.15.63:6789/0 conn(0x7f14501430c0 legacy=0x7f145014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:14.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.789+0000 7f1455e5c640 1 -- 172.21.15.63:0/1726997041 shutdown_connections 2024-03-25T18:19:14.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.789+0000 7f1455e5c640 1 -- 172.21.15.63:0/1726997041 >> 172.21.15.63:0/1726997041 conn(0x7f14500b7ea0 msgr2=0x7f14500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:14.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.789+0000 7f1455e5c640 1 -- 172.21.15.63:0/1726997041 shutdown_connections 2024-03-25T18:19:14.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.789+0000 7f1455e5c640 1 -- 172.21.15.63:0/1726997041 wait complete. 2024-03-25T18:19:14.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.789+0000 7f1455e5c640 10 monclient: build_initial_monmap 2024-03-25T18:19:14.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.789+0000 7f1455e5c640 10 monclient: monmap: 2024-03-25T18:19:14.792 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:19:14.792 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:14.792 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:19:14.792 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:19:14.792 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:14.792 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:19:14.793 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:19:14.793 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:19:14.793 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:19:14.793 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:14.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.790+0000 7f1455e5c640 1 librados: starting msgr at 2024-03-25T18:19:14.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.790+0000 7f1455e5c640 1 librados: starting objecter 2024-03-25T18:19:14.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.790+0000 7f1455e5c640 1 Processor -- start 2024-03-25T18:19:14.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.790+0000 7f1455e5c640 1 -- start start 2024-03-25T18:19:14.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.790+0000 7f1455e5c640 1 librados: setting wanted keys 2024-03-25T18:19:14.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.790+0000 7f1455e5c640 1 librados: calling monclient init 2024-03-25T18:19:14.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.790+0000 7f1455e5c640 10 monclient: init 2024-03-25T18:19:14.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.791+0000 7f1455e5c640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:14.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.791+0000 7f1455e5c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:14.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.791+0000 7f1455e5c640 10 monclient(hunting): picked mon.noname-a con 0x7f14500a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:19:14.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.791+0000 7f1455e5c640 10 monclient(hunting): picked mon.noname-c con 0x7f145013f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:19:14.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.791+0000 7f1455e5c640 10 monclient(hunting): picked mon.noname-b con 0x7f14501430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:19:14.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.791+0000 7f1455e5c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1450138bd0 con 0x7f14500a98f0 2024-03-25T18:19:14.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.791+0000 7f1455e5c640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14500aba30 con 0x7f14501430c0 2024-03-25T18:19:14.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.791+0000 7f1455e5c640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1450137740 con 0x7f145013f710 2024-03-25T18:19:14.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.791+0000 7f1455e5c640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:14.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.791+0000 7f1455e5c640 10 monclient(hunting): authenticate will time out at 7848.977694s 2024-03-25T18:19:14.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.805+0000 7f144ffff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f14501430c0 0x7f14501d7780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:38416/0 (socket says 172.21.15.63:38416) 2024-03-25T18:19:14.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.805+0000 7f144ffff640 1 -- 172.21.15.63:0/607771088 learned_addr learned my addr 172.21.15.63:0/607771088 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:19:14.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.807+0000 7f1437fff640 10 client.?.objecter ms_handle_connect 0x7f14501430c0 2024-03-25T18:19:14.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.807+0000 7f1437fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:14.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.809+0000 7f1437fff640 10 client.?.objecter ms_handle_connect 0x7f14500a98f0 2024-03-25T18:19:14.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.809+0000 7f1437fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:14.811 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.811+0000 7f1437fff640 10 client.?.objecter ms_handle_connect 0x7f145013f710 2024-03-25T18:19:14.812 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.811+0000 7f1437fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:14.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.821+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248104862 0 0) 0x7f14500aba30 con 0x7f14501430c0 2024-03-25T18:19:14.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.821+0000 7f1437fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:14.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.821+0000 7f1437fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:14.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.821+0000 7f1437fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:14.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.821+0000 7f1437fff640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:19:14.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.821+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f143c004eb0 con 0x7f14501430c0 2024-03-25T18:19:14.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.825+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265290678 0 0) 0x7f1450138bd0 con 0x7f14500a98f0 2024-03-25T18:19:14.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.825+0000 7f1437fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:14.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.825+0000 7f1437fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:14.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.825+0000 7f1437fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:14.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.825+0000 7f1437fff640 10 monclient(hunting): my global_id is 4506 2024-03-25T18:19:14.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.825+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14500aba30 con 0x7f14500a98f0 2024-03-25T18:19:14.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.826+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178647696 0 0) 0x7f1450137740 con 0x7f145013f710 2024-03-25T18:19:14.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.826+0000 7f1437fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:14.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.826+0000 7f1437fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:14.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.826+0000 7f1437fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:14.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.826+0000 7f1437fff640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:19:14.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.826+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1450138bd0 con 0x7f145013f710 2024-03-25T18:19:14.847 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.846+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2051340687 0 0) 0x7f143c004eb0 con 0x7f14501430c0 2024-03-25T18:19:14.848 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.846+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1450137740 con 0x7f14501430c0 2024-03-25T18:19:14.848 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.846+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f1448001f60 con 0x7f14501430c0 2024-03-25T18:19:14.848 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.846+0000 7f1437fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:14.848 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.846+0000 7f1437fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:14.848 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.846+0000 7f1437fff640 10 monclient(hunting): dump: 2024-03-25T18:19:14.848 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:14.848 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:14.848 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:14.848 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:14.848 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:14.848 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:14.848 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:14.848 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:14.848 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:14.848 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:14.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.846+0000 7f1437fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:14.856 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.854+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 98739495 0 0) 0x7f1450138bd0 con 0x7f145013f710 2024-03-25T18:19:14.856 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.855+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f143c004eb0 con 0x7f145013f710 2024-03-25T18:19:14.856 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.855+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f14440017b0 con 0x7f145013f710 2024-03-25T18:19:14.856 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.855+0000 7f1437fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:14.856 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.855+0000 7f1437fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:14.856 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.855+0000 7f1437fff640 10 monclient(hunting): dump: 2024-03-25T18:19:14.856 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:14.856 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:14.856 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:14.856 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:14.856 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:14.856 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:14.856 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:14.856 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:14.857 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:14.857 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:14.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3248584168 0 0) 0x7f1450137740 con 0x7f14501430c0 2024-03-25T18:19:14.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 >> v1:172.21.15.63:6790/0 conn(0x7f145013f710 legacy=0x7f1450134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:14.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 >> v1:172.21.15.63:6789/0 conn(0x7f14500a98f0 legacy=0x7f14501384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:14.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1437fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:14.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1437fff640 1 monclient: found mon.b 2024-03-25T18:19:14.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1437fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:14.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1437fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:14.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1450134de0 con 0x7f14501430c0 2024-03-25T18:19:14.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1437fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:14.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1437fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:14.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1437fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:14.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1455e5c640 5 monclient: authenticate success, global_id 4534 2024-03-25T18:19:14.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1455e5c640 10 monclient: _renew_subs 2024-03-25T18:19:14.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1455e5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:14.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1455e5c640 1 -- 172.21.15.63:0/607771088 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14500b1620 con 0x7f14501430c0 2024-03-25T18:19:14.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1455e5c640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:14.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1455e5c640 10 monclient: _renew_subs 2024-03-25T18:19:14.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1455e5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:14.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1455e5c640 1 -- 172.21.15.63:0/607771088 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1450134900 con 0x7f14501430c0 2024-03-25T18:19:14.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.863+0000 7f1455e5c640 1 librados: init done 2024-03-25T18:19:14.866 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.865+0000 7f1455e5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:14.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.865+0000 7f1455e5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:14.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.865+0000 7f1455e5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:14.867 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:14.865+0000 7f1455e5c640 1 -- 172.21.15.63:0/607771088 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14501d75d0 con 0x7f14501430c0 2024-03-25T18:19:15.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.156+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1448004e10 con 0x7f14501430c0 2024-03-25T18:19:15.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.156+0000 7f1437fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:15.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.156+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f1448005120 con 0x7f14501430c0 2024-03-25T18:19:15.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.156+0000 7f1437fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:15.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.156+0000 7f1437fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:15.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.156+0000 7f1437fff640 10 monclient: dump: 2024-03-25T18:19:15.158 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:15.158 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:15.158 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:15.158 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:15.158 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:15.158 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:15.158 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:15.158 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:15.159 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:15.159 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:15.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.156+0000 7f144dffb640 4 set_mon_vals no callback set 2024-03-25T18:19:15.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.156+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f144801b0f0 con 0x7f14501430c0 2024-03-25T18:19:15.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.157+0000 7f144f7fe640 10 monclient: get_auth_request con 0x7f143c06df70 auth_method 0 2024-03-25T18:19:15.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.157+0000 7f1437fff640 10 client.4534.objecter ms_handle_connect 0x7f143c06df70 2024-03-25T18:19:15.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.159+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f1448054c00 con 0x7f14501430c0 2024-03-25T18:19:15.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.159+0000 7f1437fff640 10 client.4534.objecter ms_dispatch 0x7f1450000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:15.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.159+0000 7f1437fff640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:15.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.159+0000 7f1437fff640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:15.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.159+0000 7f1437fff640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-25T18:19:15.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.162+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1448050710 con 0x7f14501430c0 2024-03-25T18:19:15.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.162+0000 7f1437fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:15.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.162+0000 7f1437fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:15.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.417+0000 7f1455e5c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:15.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.417+0000 7f1455e5c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:15.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.417+0000 7f1455e5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:15.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.417+0000 7f1455e5c640 1 -- 172.21.15.63:0/607771088 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14500627b0 con 0x7f14501430c0 2024-03-25T18:19:15.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.461+0000 7f1437fff640 1 -- 172.21.15.63:0/607771088 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f1448052640 con 0x7f14501430c0 2024-03-25T18:19:15.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.461+0000 7f1437fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:15.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.461+0000 7f1437fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:15.463 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:19:15.463 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:15.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.464+0000 7f1455e5c640 10 librados: watch_flush enter 2024-03-25T18:19:15.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.464+0000 7f1455e5c640 10 librados: watch_flush exit 2024-03-25T18:19:15.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.464+0000 7f1455e5c640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-25T18:19:15.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.464+0000 7f1455e5c640 10 client.4534.objecter successfully canceled tick 2024-03-25T18:19:15.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.464+0000 7f1455e5c640 1 -- 172.21.15.63:0/607771088 >> v1:172.21.15.81:6800/3640340873 conn(0x7f143c06df70 legacy=0x7f143c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.464+0000 7f1455e5c640 10 monclient: shutdown 2024-03-25T18:19:15.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.464+0000 7f1455e5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:15.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.464+0000 7f1455e5c640 1 -- 172.21.15.63:0/607771088 >> v1:172.21.15.81:6789/0 conn(0x7f14501430c0 legacy=0x7f14501d7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.464+0000 7f1455e5c640 1 -- 172.21.15.63:0/607771088 shutdown_connections 2024-03-25T18:19:15.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.465+0000 7f1455e5c640 1 -- 172.21.15.63:0/607771088 >> 172.21.15.63:0/607771088 conn(0x7f14500b7ea0 msgr2=0x7f1450074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:15.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.465+0000 7f1455e5c640 1 -- 172.21.15.63:0/607771088 shutdown_connections 2024-03-25T18:19:15.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.465+0000 7f1455e5c640 1 -- 172.21.15.63:0/607771088 wait complete. 2024-03-25T18:19:15.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:15.465+0000 7f1455e5c640 1 librados: shutdown 2024-03-25T18:19:15.478 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:19.293 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:00.985 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:19.313 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:01.006 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:19.337 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:07:00.997 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:19.348 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:01.040 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:19.351 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:07:01.011 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:19.351 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:01.043 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:19.355 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:01.047 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:19.383 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:01.075 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:19.383 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:07:01.043 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:22.284 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:22.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.394+0000 7f72163fe640 10 monclient: get_monmap_and_config 2024-03-25T18:19:22.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.394+0000 7f72163fe640 10 monclient: build_initial_monmap 2024-03-25T18:19:22.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.394+0000 7f72163fe640 10 monclient: monmap: 2024-03-25T18:19:22.396 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:19:22.396 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:22.396 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:19:22.395806+0000 2024-03-25T18:19:22.396 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:19:22.395806+0000 2024-03-25T18:19:22.396 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:22.396 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:22.396 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:19:22.396 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:19:22.396 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:19:22.396 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:22.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.395+0000 7f72163fe640 1 Processor -- start 2024-03-25T18:19:22.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.395+0000 7f72163fe640 1 -- start start 2024-03-25T18:19:22.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.395+0000 7f72163fe640 10 monclient: init 2024-03-25T18:19:22.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.396+0000 7f72163fe640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:22.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.396+0000 7f72163fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:22.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.396+0000 7f72163fe640 10 monclient(hunting): picked mon.noname-c con 0x7f72100a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:19:22.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.396+0000 7f72163fe640 10 monclient(hunting): picked mon.noname-a con 0x7f721013f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:19:22.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.396+0000 7f72163fe640 10 monclient(hunting): picked mon.noname-b con 0x7f72101430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:19:22.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.396+0000 7f72163fe640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7210142520 con 0x7f721013f710 2024-03-25T18:19:22.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.396+0000 7f72163fe640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72100b28a0 con 0x7f72100a98f0 2024-03-25T18:19:22.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.396+0000 7f72163fe640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72100b2a30 con 0x7f72101430c0 2024-03-25T18:19:22.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.396+0000 7f72163fe640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:22.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.396+0000 7f72163fe640 10 monclient(hunting): authenticate will time out at 7856.582569s 2024-03-25T18:19:22.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.401+0000 7f720ffff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f72100a98f0 0x7f72100a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:40950/0 (socket says 172.21.15.63:40950) 2024-03-25T18:19:22.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.401+0000 7f720ffff640 1 -- 172.21.15.63:0/669537964 learned_addr learned my addr 172.21.15.63:0/669537964 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:19:22.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.425+0000 7f720effd640 1 -- 172.21.15.63:0/669537964 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151594646 0 0) 0x7f72100b28a0 con 0x7f72100a98f0 2024-03-25T18:19:22.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.425+0000 7f720effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:22.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.425+0000 7f720effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:22.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.425+0000 7f720effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:22.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.425+0000 7f720effd640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:19:22.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.425+0000 7f720effd640 1 -- 172.21.15.63:0/669537964 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71f8003520 con 0x7f72100a98f0 2024-03-25T18:19:22.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.425+0000 7f720effd640 1 -- 172.21.15.63:0/669537964 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013059613 0 0) 0x7f7210142520 con 0x7f721013f710 2024-03-25T18:19:22.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.425+0000 7f720effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:22.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.425+0000 7f720effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:22.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.425+0000 7f720effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:22.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.425+0000 7f720effd640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:19:22.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.425+0000 7f720effd640 1 -- 172.21.15.63:0/669537964 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72100b28a0 con 0x7f721013f710 2024-03-25T18:19:22.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.430+0000 7f720effd640 1 -- 172.21.15.63:0/669537964 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582014243 0 0) 0x7f72100b2a30 con 0x7f72101430c0 2024-03-25T18:19:22.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.430+0000 7f720effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:22.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.430+0000 7f720effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:22.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.430+0000 7f720effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:22.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.430+0000 7f720effd640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:19:22.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.430+0000 7f720effd640 1 -- 172.21.15.63:0/669537964 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7210142520 con 0x7f72101430c0 2024-03-25T18:19:22.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.440+0000 7f720effd640 1 -- 172.21.15.63:0/669537964 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 736434454 0 0) 0x7f7210142520 con 0x7f72101430c0 2024-03-25T18:19:22.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.440+0000 7f720effd640 1 -- 172.21.15.63:0/669537964 >> v1:172.21.15.81:6789/0 conn(0x7f72100a98f0 legacy=0x7f72100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:22.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.440+0000 7f720effd640 1 -- 172.21.15.63:0/669537964 >> v1:172.21.15.63:6789/0 conn(0x7f721013f710 legacy=0x7f72101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:22.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.440+0000 7f720effd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:22.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.440+0000 7f720effd640 1 monclient: found mon.noname-b 2024-03-25T18:19:22.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.440+0000 7f720effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:22.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.440+0000 7f720effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:19:22.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.440+0000 7f720effd640 1 -- 172.21.15.63:0/669537964 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72100b44c0 con 0x7f72101430c0 2024-03-25T18:19:22.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.440+0000 7f720effd640 10 monclient: _finish_auth 0 2024-03-25T18:19:22.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.440+0000 7f720effd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.440+0000 7f720effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.440+0000 7f720effd640 1 -- 172.21.15.63:0/669537964 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7208001930 con 0x7f72101430c0 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.440+0000 7f720effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.440+0000 7f720effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.440+0000 7f720effd640 10 monclient: dump: 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:22.444 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:22.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.441+0000 7f72163fe640 5 monclient: authenticate success, global_id 4514 2024-03-25T18:19:22.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.441+0000 7f72163fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:22.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.452+0000 7f720effd640 1 -- 172.21.15.63:0/669537964 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7208001a90 con 0x7f72101430c0 2024-03-25T18:19:22.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.452+0000 7f720effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:22.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.452+0000 7f720effd640 1 -- 172.21.15.63:0/669537964 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7208001ee0 con 0x7f72101430c0 2024-03-25T18:19:22.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.452+0000 7f72163fe640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:22.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.452+0000 7f72163fe640 4 set_mon_vals no callback set 2024-03-25T18:19:22.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.452+0000 7f720effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:22.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.452+0000 7f720effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:22.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.452+0000 7f720effd640 10 monclient: dump: 2024-03-25T18:19:22.454 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:22.454 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:22.454 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:22.454 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:22.454 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:22.454 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:22.455 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:22.455 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:22.455 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:22.455 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:22.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.453+0000 7f72163fe640 10 monclient: shutdown 2024-03-25T18:19:22.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.453+0000 7f72163fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:22.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.453+0000 7f72163fe640 1 -- 172.21.15.63:0/669537964 >> v1:172.21.15.63:6790/0 conn(0x7f72101430c0 legacy=0x7f721014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:22.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.453+0000 7f72163fe640 1 -- 172.21.15.63:0/669537964 shutdown_connections 2024-03-25T18:19:22.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.453+0000 7f72163fe640 1 -- 172.21.15.63:0/669537964 >> 172.21.15.63:0/669537964 conn(0x7f72100b7ea0 msgr2=0x7f72100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:22.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.453+0000 7f72163fe640 1 -- 172.21.15.63:0/669537964 shutdown_connections 2024-03-25T18:19:22.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.453+0000 7f72163fe640 1 -- 172.21.15.63:0/669537964 wait complete. 2024-03-25T18:19:22.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.454+0000 7f72163fe640 10 monclient: build_initial_monmap 2024-03-25T18:19:22.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.454+0000 7f72163fe640 10 monclient: monmap: 2024-03-25T18:19:22.457 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:19:22.457 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:22.457 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:19:22.458 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:19:22.458 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:22.458 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:19:22.458 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:19:22.458 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:19:22.458 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:19:22.458 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:22.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.454+0000 7f72163fe640 1 librados: starting msgr at 2024-03-25T18:19:22.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.454+0000 7f72163fe640 1 librados: starting objecter 2024-03-25T18:19:22.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.454+0000 7f72163fe640 1 Processor -- start 2024-03-25T18:19:22.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.455+0000 7f72163fe640 1 -- start start 2024-03-25T18:19:22.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.455+0000 7f72163fe640 1 librados: setting wanted keys 2024-03-25T18:19:22.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.455+0000 7f72163fe640 1 librados: calling monclient init 2024-03-25T18:19:22.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.455+0000 7f72163fe640 10 monclient: init 2024-03-25T18:19:22.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.455+0000 7f72163fe640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:22.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.455+0000 7f72163fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:22.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.455+0000 7f72163fe640 10 monclient(hunting): picked mon.noname-c con 0x7f72100a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:19:22.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.455+0000 7f72163fe640 10 monclient(hunting): picked mon.noname-a con 0x7f721013f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:19:22.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.455+0000 7f72163fe640 10 monclient(hunting): picked mon.noname-b con 0x7f72101430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:19:22.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.455+0000 7f72163fe640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72101dae50 con 0x7f721013f710 2024-03-25T18:19:22.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.455+0000 7f72163fe640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7210138ee0 con 0x7f72101430c0 2024-03-25T18:19:22.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.455+0000 7f72163fe640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72101daa20 con 0x7f72100a98f0 2024-03-25T18:19:22.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.455+0000 7f72163fe640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:22.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.455+0000 7f72163fe640 10 monclient(hunting): authenticate will time out at 7856.642161s 2024-03-25T18:19:22.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.467+0000 7f720ffff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f72100a98f0 0x7f72101db2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:55770/0 (socket says 172.21.15.63:55770) 2024-03-25T18:19:22.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.467+0000 7f720ffff640 1 -- 172.21.15.63:0/1596032601 learned_addr learned my addr 172.21.15.63:0/1596032601 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:19:22.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.471+0000 7f720cff9640 10 client.?.objecter ms_handle_connect 0x7f72100a98f0 2024-03-25T18:19:22.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.471+0000 7f720cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:22.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.473+0000 7f720cff9640 10 client.?.objecter ms_handle_connect 0x7f721013f710 2024-03-25T18:19:22.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.473+0000 7f720cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:22.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.478+0000 7f720cff9640 10 client.?.objecter ms_handle_connect 0x7f72101430c0 2024-03-25T18:19:22.480 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.478+0000 7f720cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:22.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.481+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892102157 0 0) 0x7f72101dae50 con 0x7f721013f710 2024-03-25T18:19:22.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.481+0000 7f720cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:22.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.481+0000 7f720cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:22.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.481+0000 7f720cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:22.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.481+0000 7f720cff9640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:19:22.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.481+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71fc004eb0 con 0x7f721013f710 2024-03-25T18:19:22.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.482+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417522117 0 0) 0x7f72101daa20 con 0x7f72100a98f0 2024-03-25T18:19:22.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.482+0000 7f720cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:22.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.482+0000 7f720cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:22.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.482+0000 7f720cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:22.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.482+0000 7f720cff9640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:19:22.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.482+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72101dae50 con 0x7f72100a98f0 2024-03-25T18:19:22.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.493+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754213567 0 0) 0x7f7210138ee0 con 0x7f72101430c0 2024-03-25T18:19:22.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.493+0000 7f720cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:22.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.493+0000 7f720cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:22.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.493+0000 7f720cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:22.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.493+0000 7f720cff9640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:19:22.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.493+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72101daa20 con 0x7f72101430c0 2024-03-25T18:19:22.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.493+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1044425877 0 0) 0x7f72101dae50 con 0x7f72100a98f0 2024-03-25T18:19:22.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.493+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7210138ee0 con 0x7f72100a98f0 2024-03-25T18:19:22.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.493+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f72000018d0 con 0x7f72100a98f0 2024-03-25T18:19:22.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.493+0000 7f720cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:22.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.493+0000 7f720cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:22.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.493+0000 7f720cff9640 10 monclient(hunting): dump: 2024-03-25T18:19:22.495 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:22.495 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:22.495 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:22.496 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:22.496 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:22.496 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:22.496 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:22.496 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:22.496 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:22.496 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:22.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.493+0000 7f720cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:22.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.500+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3599602299 0 0) 0x7f72101daa20 con 0x7f72101430c0 2024-03-25T18:19:22.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.500+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72101dae50 con 0x7f72101430c0 2024-03-25T18:19:22.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.500+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7208001da0 con 0x7f72101430c0 2024-03-25T18:19:22.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.500+0000 7f720cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:22.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.500+0000 7f720cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:22.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.500+0000 7f720cff9640 10 monclient(hunting): dump: 2024-03-25T18:19:22.502 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:22.502 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:22.502 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:22.502 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:22.502 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:22.502 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:22.502 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:22.502 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:22.502 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:22.502 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:22.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 90310475 0 0) 0x7f7210138ee0 con 0x7f72100a98f0 2024-03-25T18:19:22.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 >> v1:172.21.15.81:6789/0 conn(0x7f72101430c0 legacy=0x7f72101e2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:22.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 >> v1:172.21.15.63:6789/0 conn(0x7f721013f710 legacy=0x7f72101387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:22.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f720cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:22.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f720cff9640 1 monclient: found mon.c 2024-03-25T18:19:22.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f720cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:22.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f720cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:19:22.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7210137f80 con 0x7f72100a98f0 2024-03-25T18:19:22.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f720cff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:22.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f720cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:22.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f720cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:22.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f72163fe640 5 monclient: authenticate success, global_id 4517 2024-03-25T18:19:22.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f72163fe640 10 monclient: _renew_subs 2024-03-25T18:19:22.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f72163fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:19:22.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f72163fe640 1 -- 172.21.15.63:0/1596032601 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72101381d0 con 0x7f72100a98f0 2024-03-25T18:19:22.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f72163fe640 10 client.4517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:22.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f72163fe640 10 monclient: _renew_subs 2024-03-25T18:19:22.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f72163fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:19:22.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f72163fe640 1 -- 172.21.15.63:0/1596032601 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72101352f0 con 0x7f72100a98f0 2024-03-25T18:19:22.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.505+0000 7f72163fe640 1 librados: init done 2024-03-25T18:19:22.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.506+0000 7f72163fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:22.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.506+0000 7f72163fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:22.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.506+0000 7f72163fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:19:22.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.506+0000 7f72163fe640 1 -- 172.21.15.63:0/1596032601 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72100b3ef0 con 0x7f72100a98f0 2024-03-25T18:19:22.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.724+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72000047e0 con 0x7f72100a98f0 2024-03-25T18:19:22.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.724+0000 7f720cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:22.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.724+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7200004af0 con 0x7f72100a98f0 2024-03-25T18:19:22.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.724+0000 7f720cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:22.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.724+0000 7f720cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:22.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.724+0000 7f720cff9640 10 monclient: dump: 2024-03-25T18:19:22.725 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:22.725 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:22.725 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:22.725 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:22.725 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:22.726 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:22.726 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:22.726 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:22.726 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:22.726 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:22.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.724+0000 7f720e7fc640 4 set_mon_vals no callback set 2024-03-25T18:19:22.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.816+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f720001aac0 con 0x7f72100a98f0 2024-03-25T18:19:22.818 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.817+0000 7f720f7fe640 10 monclient: get_auth_request con 0x7f71fc06df80 auth_method 0 2024-03-25T18:19:22.818 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.817+0000 7f720cff9640 10 client.4517.objecter ms_handle_connect 0x7f71fc06df80 2024-03-25T18:19:22.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.824+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f7200054750 con 0x7f72100a98f0 2024-03-25T18:19:22.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.824+0000 7f720cff9640 10 client.4517.objecter ms_dispatch 0x7f7210000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:22.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.824+0000 7f720cff9640 3 client.4517.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:22.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.824+0000 7f720cff9640 3 client.4517.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:22.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.824+0000 7f720cff9640 20 client.4517.objecter dump_active .. 0 homeless 2024-03-25T18:19:22.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.824+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7200050090 con 0x7f72100a98f0 2024-03-25T18:19:22.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.824+0000 7f720cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:22.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:22.824+0000 7f720cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:23.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.058+0000 7f72163fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:23.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.058+0000 7f72163fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:23.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.058+0000 7f72163fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:19:23.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.058+0000 7f72163fe640 1 -- 172.21.15.63:0/1596032601 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72101dabb0 con 0x7f72100a98f0 2024-03-25T18:19:23.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.089+0000 7f720cff9640 1 -- 172.21.15.63:0/1596032601 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f7200052190 con 0x7f72100a98f0 2024-03-25T18:19:23.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.089+0000 7f720cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:23.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.089+0000 7f720cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:23.091 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:19:23.091 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:23.093 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.092+0000 7f72163fe640 10 librados: watch_flush enter 2024-03-25T18:19:23.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.092+0000 7f72163fe640 10 librados: watch_flush exit 2024-03-25T18:19:23.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.092+0000 7f72163fe640 20 client.4517.objecter shutdown clearing up homeless session... 2024-03-25T18:19:23.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.092+0000 7f72163fe640 10 client.4517.objecter successfully canceled tick 2024-03-25T18:19:23.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.092+0000 7f72163fe640 1 -- 172.21.15.63:0/1596032601 >> v1:172.21.15.81:6800/3640340873 conn(0x7f71fc06df80 legacy=0x7f71fc070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.092+0000 7f72163fe640 10 monclient: shutdown 2024-03-25T18:19:23.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.092+0000 7f72163fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:23.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.092+0000 7f72163fe640 1 -- 172.21.15.63:0/1596032601 >> v1:172.21.15.63:6790/0 conn(0x7f72100a98f0 legacy=0x7f72101db2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.092+0000 7f72163fe640 1 -- 172.21.15.63:0/1596032601 shutdown_connections 2024-03-25T18:19:23.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.092+0000 7f72163fe640 1 -- 172.21.15.63:0/1596032601 >> 172.21.15.63:0/1596032601 conn(0x7f72100b7ea0 msgr2=0x7f7210074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:23.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.092+0000 7f72163fe640 1 -- 172.21.15.63:0/1596032601 shutdown_connections 2024-03-25T18:19:23.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.092+0000 7f72163fe640 1 -- 172.21.15.63:0/1596032601 wait complete. 2024-03-25T18:19:23.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:23.092+0000 7f72163fe640 1 librados: shutdown 2024-03-25T18:19:23.107 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:29.913 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:30.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.028+0000 7f987326a640 10 monclient: get_monmap_and_config 2024-03-25T18:19:30.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.028+0000 7f987326a640 10 monclient: build_initial_monmap 2024-03-25T18:19:30.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.028+0000 7f987326a640 10 monclient: monmap: 2024-03-25T18:19:30.030 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:19:30.030 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:30.030 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:19:30.029703+0000 2024-03-25T18:19:30.030 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:19:30.029703+0000 2024-03-25T18:19:30.030 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:30.030 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:30.030 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:19:30.030 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:19:30.030 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:19:30.030 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:30.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.029+0000 7f987326a640 1 Processor -- start 2024-03-25T18:19:30.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.029+0000 7f987326a640 1 -- start start 2024-03-25T18:19:30.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.029+0000 7f987326a640 10 monclient: init 2024-03-25T18:19:30.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.030+0000 7f987326a640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:30.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.030+0000 7f987326a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:30.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.030+0000 7f987326a640 10 monclient(hunting): picked mon.noname-a con 0x7f986c13f990 addr v1:172.21.15.63:6789/0 2024-03-25T18:19:30.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.030+0000 7f987326a640 10 monclient(hunting): picked mon.noname-c con 0x7f986c13fd70 addr v1:172.21.15.81:6789/0 2024-03-25T18:19:30.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.030+0000 7f987326a640 10 monclient(hunting): picked mon.noname-b con 0x7f986c0afa60 addr v1:172.21.15.63:6790/0 2024-03-25T18:19:30.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.030+0000 7f987326a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986c0afeb0 con 0x7f986c13f990 2024-03-25T18:19:30.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.030+0000 7f987326a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986c0b0100 con 0x7f986c13fd70 2024-03-25T18:19:30.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.030+0000 7f987326a640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986c0b1150 con 0x7f986c0afa60 2024-03-25T18:19:30.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.030+0000 7f987326a640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:30.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.030+0000 7f987326a640 10 monclient(hunting): authenticate will time out at 7864.216954s 2024-03-25T18:19:30.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.035+0000 7f9871807640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f986c0afa60 0x7f986c14dbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:51542/0 (socket says 172.21.15.63:51542) 2024-03-25T18:19:30.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.035+0000 7f9871807640 1 -- 172.21.15.63:0/1717184978 learned_addr learned my addr 172.21.15.63:0/1717184978 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:19:30.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.048+0000 7f9863fff640 1 -- 172.21.15.63:0/1717184978 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233694145 0 0) 0x7f986c0b1150 con 0x7f986c0afa60 2024-03-25T18:19:30.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.048+0000 7f9863fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:30.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.048+0000 7f9863fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:30.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.048+0000 7f9863fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:30.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.048+0000 7f9863fff640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:19:30.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.048+0000 7f9863fff640 1 -- 172.21.15.63:0/1717184978 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9858003520 con 0x7f986c0afa60 2024-03-25T18:19:30.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f9863fff640 1 -- 172.21.15.63:0/1717184978 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2789812405 0 0) 0x7f9858003520 con 0x7f986c0afa60 2024-03-25T18:19:30.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f9863fff640 1 -- 172.21.15.63:0/1717184978 >> v1:172.21.15.81:6789/0 conn(0x7f986c13fd70 legacy=0x7f986c145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f9863fff640 1 -- 172.21.15.63:0/1717184978 >> v1:172.21.15.63:6789/0 conn(0x7f986c13f990 legacy=0x7f986c06ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f9863fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:30.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f9863fff640 1 monclient: found mon.noname-b 2024-03-25T18:19:30.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f9863fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:30.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f9863fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:19:30.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f9863fff640 1 -- 172.21.15.63:0/1717184978 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c0aede0 con 0x7f986c0afa60 2024-03-25T18:19:30.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f9863fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:30.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f9863fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:30.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f9863fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:30.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f9863fff640 1 -- 172.21.15.63:0/1717184978 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9864001a10 con 0x7f986c0afa60 2024-03-25T18:19:30.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f987326a640 5 monclient: authenticate success, global_id 4520 2024-03-25T18:19:30.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f987326a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:30.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f9863fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:30.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f9863fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:30.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f9863fff640 10 monclient: dump: 2024-03-25T18:19:30.057 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:30.057 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:30.057 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:30.057 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:30.057 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:30.057 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:30.057 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:30.057 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:30.057 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:30.057 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:30.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.054+0000 7f987326a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:30.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.063+0000 7f9863fff640 1 -- 172.21.15.63:0/1717184978 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9864001b70 con 0x7f986c0afa60 2024-03-25T18:19:30.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.063+0000 7f9863fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:30.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.063+0000 7f9863fff640 1 -- 172.21.15.63:0/1717184978 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9864001f80 con 0x7f986c0afa60 2024-03-25T18:19:30.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.063+0000 7f987326a640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:30.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.063+0000 7f987326a640 4 set_mon_vals no callback set 2024-03-25T18:19:30.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.063+0000 7f9863fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:30.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.063+0000 7f9863fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:30.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.063+0000 7f9863fff640 10 monclient: dump: 2024-03-25T18:19:30.065 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:30.065 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:30.065 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:30.065 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:30.065 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:30.065 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:30.065 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:30.065 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:30.066 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:30.066 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:30.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.064+0000 7f987326a640 10 monclient: shutdown 2024-03-25T18:19:30.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.064+0000 7f987326a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:30.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.064+0000 7f987326a640 1 -- 172.21.15.63:0/1717184978 >> v1:172.21.15.63:6790/0 conn(0x7f986c0afa60 legacy=0x7f986c14dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.064+0000 7f987326a640 1 -- 172.21.15.63:0/1717184978 shutdown_connections 2024-03-25T18:19:30.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.064+0000 7f987326a640 1 -- 172.21.15.63:0/1717184978 >> 172.21.15.63:0/1717184978 conn(0x7f986c0b4b80 msgr2=0x7f986c0b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:30.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.064+0000 7f987326a640 1 -- 172.21.15.63:0/1717184978 shutdown_connections 2024-03-25T18:19:30.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.064+0000 7f987326a640 1 -- 172.21.15.63:0/1717184978 wait complete. 2024-03-25T18:19:30.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.065+0000 7f987326a640 10 monclient: build_initial_monmap 2024-03-25T18:19:30.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.065+0000 7f987326a640 10 monclient: monmap: 2024-03-25T18:19:30.069 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:19:30.069 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:30.069 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:19:30.069 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:19:30.069 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:30.069 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:19:30.069 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:19:30.069 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:19:30.069 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:19:30.069 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:30.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.065+0000 7f987326a640 1 librados: starting msgr at 2024-03-25T18:19:30.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.065+0000 7f987326a640 1 librados: starting objecter 2024-03-25T18:19:30.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.065+0000 7f987326a640 1 Processor -- start 2024-03-25T18:19:30.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.066+0000 7f987326a640 1 -- start start 2024-03-25T18:19:30.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.066+0000 7f987326a640 1 librados: setting wanted keys 2024-03-25T18:19:30.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.066+0000 7f987326a640 1 librados: calling monclient init 2024-03-25T18:19:30.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.066+0000 7f987326a640 10 monclient: init 2024-03-25T18:19:30.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.066+0000 7f987326a640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:30.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.066+0000 7f987326a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:30.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.066+0000 7f987326a640 10 monclient(hunting): picked mon.noname-b con 0x7f986c0afa60 addr v1:172.21.15.81:6789/0 2024-03-25T18:19:30.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.066+0000 7f987326a640 10 monclient(hunting): picked mon.noname-a con 0x7f986c13f990 addr v1:172.21.15.63:6789/0 2024-03-25T18:19:30.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.066+0000 7f987326a640 10 monclient(hunting): picked mon.noname-c con 0x7f986c13fd70 addr v1:172.21.15.63:6790/0 2024-03-25T18:19:30.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.066+0000 7f987326a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986c11d1e0 con 0x7f986c13f990 2024-03-25T18:19:30.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.066+0000 7f987326a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986c1d4eb0 con 0x7f986c0afa60 2024-03-25T18:19:30.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.066+0000 7f987326a640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986c11c760 con 0x7f986c13fd70 2024-03-25T18:19:30.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.066+0000 7f987326a640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:30.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.066+0000 7f987326a640 10 monclient(hunting): authenticate will time out at 7864.253040s 2024-03-25T18:19:30.082 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.081+0000 7f9870805640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f986c13f990 0x7f986c11d670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:43564/0 (socket says 172.21.15.63:43564) 2024-03-25T18:19:30.082 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.081+0000 7f9870805640 1 -- 172.21.15.63:0/2702613541 learned_addr learned my addr 172.21.15.63:0/2702613541 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:19:30.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.083+0000 7f9861ffb640 10 client.?.objecter ms_handle_connect 0x7f986c13f990 2024-03-25T18:19:30.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.083+0000 7f9861ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:30.093 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.092+0000 7f9861ffb640 10 client.?.objecter ms_handle_connect 0x7f986c0afa60 2024-03-25T18:19:30.093 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.092+0000 7f9861ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:30.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.097+0000 7f9861ffb640 10 client.?.objecter ms_handle_connect 0x7f986c13fd70 2024-03-25T18:19:30.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.097+0000 7f9861ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:30.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.098+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009023211 0 0) 0x7f986c11d1e0 con 0x7f986c13f990 2024-03-25T18:19:30.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.098+0000 7f9861ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:30.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.098+0000 7f9861ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:30.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.098+0000 7f9861ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:30.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.098+0000 7f9861ffb640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:19:30.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.098+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9850004eb0 con 0x7f986c13f990 2024-03-25T18:19:30.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.107+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202710061 0 0) 0x7f986c1d4eb0 con 0x7f986c0afa60 2024-03-25T18:19:30.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.107+0000 7f9861ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:30.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.107+0000 7f9861ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:30.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.107+0000 7f9861ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:30.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.107+0000 7f9861ffb640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:19:30.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.107+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f986c11d1e0 con 0x7f986c0afa60 2024-03-25T18:19:30.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.113+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078747491 0 0) 0x7f986c11c760 con 0x7f986c13fd70 2024-03-25T18:19:30.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.113+0000 7f9861ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:30.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.113+0000 7f9861ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:30.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.114+0000 7f9861ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:30.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.114+0000 7f9861ffb640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:19:30.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.114+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f986c1d4eb0 con 0x7f986c13fd70 2024-03-25T18:19:30.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.117+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2950607069 0 0) 0x7f9850004eb0 con 0x7f986c13f990 2024-03-25T18:19:30.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.117+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f986c11c760 con 0x7f986c13f990 2024-03-25T18:19:30.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.117+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f985c001990 con 0x7f986c13f990 2024-03-25T18:19:30.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.117+0000 7f9861ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:30.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.117+0000 7f9861ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:30.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.117+0000 7f9861ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:30.119 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:30.119 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:30.119 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:30.119 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:30.119 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:30.119 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:30.119 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:30.119 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:30.119 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:30.119 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:30.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.117+0000 7f9861ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:30.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.117+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2227149818 0 0) 0x7f986c11d1e0 con 0x7f986c0afa60 2024-03-25T18:19:30.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.117+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9850004eb0 con 0x7f986c0afa60 2024-03-25T18:19:30.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.117+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f98680018a0 con 0x7f986c0afa60 2024-03-25T18:19:30.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.117+0000 7f9861ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:30.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.117+0000 7f9861ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:30.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.117+0000 7f9861ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:30.120 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:30.120 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:30.120 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:30.120 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:30.120 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:30.120 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:30.120 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:30.120 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:30.120 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:30.120 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:30.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.127+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 593316809 0 0) 0x7f9850004eb0 con 0x7f986c0afa60 2024-03-25T18:19:30.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 >> v1:172.21.15.63:6790/0 conn(0x7f986c13fd70 legacy=0x7f986c1e2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 >> v1:172.21.15.63:6789/0 conn(0x7f986c13f990 legacy=0x7f986c11d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f9861ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:30.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f9861ffb640 1 monclient: found mon.b 2024-03-25T18:19:30.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f9861ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:30.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f9861ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:30.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c1db510 con 0x7f986c0afa60 2024-03-25T18:19:30.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f9861ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:30.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f9861ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:30.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f9861ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:30.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f987326a640 5 monclient: authenticate success, global_id 4546 2024-03-25T18:19:30.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f987326a640 10 monclient: _renew_subs 2024-03-25T18:19:30.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f987326a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:30.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f987326a640 1 -- 172.21.15.63:0/2702613541 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f986c1db700 con 0x7f986c0afa60 2024-03-25T18:19:30.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f987326a640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:30.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f987326a640 10 monclient: _renew_subs 2024-03-25T18:19:30.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f987326a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:30.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f987326a640 1 -- 172.21.15.63:0/2702613541 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f986c11cea0 con 0x7f986c0afa60 2024-03-25T18:19:30.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.128+0000 7f987326a640 1 librados: init done 2024-03-25T18:19:30.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.129+0000 7f987326a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:30.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.129+0000 7f987326a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:30.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.129+0000 7f987326a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:30.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.129+0000 7f987326a640 1 -- 172.21.15.63:0/2702613541 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f986c06bac0 con 0x7f986c0afa60 2024-03-25T18:19:30.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.414+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9868001d50 con 0x7f986c0afa60 2024-03-25T18:19:30.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.414+0000 7f9861ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:30.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.414+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f98680037d0 con 0x7f986c0afa60 2024-03-25T18:19:30.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.414+0000 7f9861ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:30.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.414+0000 7f9861ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:30.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.414+0000 7f9861ffb640 10 monclient: dump: 2024-03-25T18:19:30.416 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:30.416 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:30.416 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:30.416 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:30.416 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:30.417 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:30.417 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:30.417 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:30.417 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:30.417 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:30.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.414+0000 7f98637fe640 4 set_mon_vals no callback set 2024-03-25T18:19:30.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.415+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f98680039f0 con 0x7f986c0afa60 2024-03-25T18:19:30.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.416+0000 7f9870805640 10 monclient: get_auth_request con 0x7f985006df70 auth_method 0 2024-03-25T18:19:30.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.416+0000 7f9861ffb640 10 client.4546.objecter ms_handle_connect 0x7f985006df70 2024-03-25T18:19:30.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.421+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f9868053510 con 0x7f986c0afa60 2024-03-25T18:19:30.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.421+0000 7f9861ffb640 10 client.4546.objecter ms_dispatch 0x7f986c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:30.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.421+0000 7f9861ffb640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:30.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.421+0000 7f9861ffb640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:30.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.421+0000 7f9861ffb640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T18:19:30.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.421+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f986804ee50 con 0x7f986c0afa60 2024-03-25T18:19:30.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.421+0000 7f9861ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:30.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.421+0000 7f9861ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:30.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.654+0000 7f987326a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:30.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.654+0000 7f987326a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:30.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.654+0000 7f987326a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:30.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.654+0000 7f987326a640 1 -- 172.21.15.63:0/2702613541 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f986c0627b0 con 0x7f986c0afa60 2024-03-25T18:19:30.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.673+0000 7f9861ffb640 1 -- 172.21.15.63:0/2702613541 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f9868050f50 con 0x7f986c0afa60 2024-03-25T18:19:30.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.673+0000 7f9861ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:30.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.673+0000 7f9861ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:30.675 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:19:30.675 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:30.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.676+0000 7f987326a640 10 librados: watch_flush enter 2024-03-25T18:19:30.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.676+0000 7f987326a640 10 librados: watch_flush exit 2024-03-25T18:19:30.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.676+0000 7f987326a640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T18:19:30.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.676+0000 7f987326a640 10 client.4546.objecter successfully canceled tick 2024-03-25T18:19:30.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.676+0000 7f987326a640 1 -- 172.21.15.63:0/2702613541 >> v1:172.21.15.81:6800/3640340873 conn(0x7f985006df70 legacy=0x7f9850070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.676+0000 7f987326a640 10 monclient: shutdown 2024-03-25T18:19:30.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.676+0000 7f987326a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:30.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.676+0000 7f987326a640 1 -- 172.21.15.63:0/2702613541 >> v1:172.21.15.81:6789/0 conn(0x7f986c0afa60 legacy=0x7f986c13bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.676+0000 7f987326a640 1 -- 172.21.15.63:0/2702613541 shutdown_connections 2024-03-25T18:19:30.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.676+0000 7f987326a640 1 -- 172.21.15.63:0/2702613541 >> 172.21.15.63:0/2702613541 conn(0x7f986c0b4b80 msgr2=0x7f986c06f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:30.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.676+0000 7f987326a640 1 -- 172.21.15.63:0/2702613541 shutdown_connections 2024-03-25T18:19:30.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.676+0000 7f987326a640 1 -- 172.21.15.63:0/2702613541 wait complete. 2024-03-25T18:19:30.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:30.676+0000 7f987326a640 1 librados: shutdown 2024-03-25T18:19:30.688 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:37.496 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:37.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.607+0000 7f1944c0f640 10 monclient: get_monmap_and_config 2024-03-25T18:19:37.609 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.607+0000 7f1944c0f640 10 monclient: build_initial_monmap 2024-03-25T18:19:37.609 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.607+0000 7f1944c0f640 10 monclient: monmap: 2024-03-25T18:19:37.609 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:19:37.609 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:37.609 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:19:37.608908+0000 2024-03-25T18:19:37.609 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:19:37.608908+0000 2024-03-25T18:19:37.609 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:37.609 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:37.609 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:19:37.609 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:19:37.610 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:19:37.610 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:37.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.608+0000 7f1944c0f640 1 Processor -- start 2024-03-25T18:19:37.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.608+0000 7f1944c0f640 1 -- start start 2024-03-25T18:19:37.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.608+0000 7f1944c0f640 10 monclient: init 2024-03-25T18:19:37.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.609+0000 7f1944c0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:37.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.609+0000 7f1944c0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:37.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.609+0000 7f1944c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f19400a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:19:37.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.609+0000 7f1944c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f194013f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:19:37.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.609+0000 7f1944c0f640 10 monclient(hunting): picked mon.noname-a con 0x7f19401430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:19:37.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.609+0000 7f1944c0f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1940142520 con 0x7f19401430c0 2024-03-25T18:19:37.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.609+0000 7f1944c0f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19400b28a0 con 0x7f194013f710 2024-03-25T18:19:37.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.609+0000 7f1944c0f640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19400b2a30 con 0x7f19400a98f0 2024-03-25T18:19:37.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.609+0000 7f1944c0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:37.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.609+0000 7f1944c0f640 10 monclient(hunting): authenticate will time out at 7871.795836s 2024-03-25T18:19:37.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.622+0000 7f193e59c640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f19400a98f0 0x7f19400a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:56352/0 (socket says 172.21.15.63:56352) 2024-03-25T18:19:37.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.622+0000 7f193ed9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f19401430c0 0x7f194014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48288/0 (socket says 172.21.15.63:48288) 2024-03-25T18:19:37.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.622+0000 7f193e59c640 1 -- 172.21.15.63:0/2595387038 learned_addr learned my addr 172.21.15.63:0/2595387038 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:19:37.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.629+0000 7f193dd9b640 1 -- 172.21.15.63:0/2595387038 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779515650 0 0) 0x7f1940142520 con 0x7f19401430c0 2024-03-25T18:19:37.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.629+0000 7f193dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:37.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.629+0000 7f193dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:37.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.629+0000 7f193dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:37.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.629+0000 7f193dd9b640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:19:37.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.629+0000 7f193dd9b640 1 -- 172.21.15.63:0/2595387038 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f192c003520 con 0x7f19401430c0 2024-03-25T18:19:37.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.640+0000 7f193dd9b640 1 -- 172.21.15.63:0/2595387038 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014395728 0 0) 0x7f19400b2a30 con 0x7f19400a98f0 2024-03-25T18:19:37.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.640+0000 7f193dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:37.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.640+0000 7f193dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:37.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.640+0000 7f193dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:37.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.640+0000 7f193dd9b640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:19:37.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.640+0000 7f193dd9b640 1 -- 172.21.15.63:0/2595387038 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1940142520 con 0x7f19400a98f0 2024-03-25T18:19:37.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f193dd9b640 1 -- 172.21.15.63:0/2595387038 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 713536597 0 0) 0x7f192c003520 con 0x7f19401430c0 2024-03-25T18:19:37.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f193dd9b640 1 -- 172.21.15.63:0/2595387038 >> v1:172.21.15.63:6790/0 conn(0x7f19400a98f0 legacy=0x7f19400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:37.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f193dd9b640 1 -- 172.21.15.63:0/2595387038 >> v1:172.21.15.81:6789/0 conn(0x7f194013f710 legacy=0x7f19401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:37.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f193dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:19:37.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f193dd9b640 1 monclient: found mon.noname-a 2024-03-25T18:19:37.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f193dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:37.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f193dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:19:37.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f193dd9b640 1 -- 172.21.15.63:0/2595387038 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19400b44c0 con 0x7f19401430c0 2024-03-25T18:19:37.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f193dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:19:37.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f193dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:19:37.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f193dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:37.647 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f193dd9b640 1 -- 172.21.15.63:0/2595387038 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f1938001a10 con 0x7f19401430c0 2024-03-25T18:19:37.647 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f193dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:37.647 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f193dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:37.647 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f193dd9b640 10 monclient: dump: 2024-03-25T18:19:37.647 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:37.647 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:37.647 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:37.647 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:37.647 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:37.647 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:37.647 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:37.647 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:37.648 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:37.648 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:37.648 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f1944c0f640 5 monclient: authenticate success, global_id 4521 2024-03-25T18:19:37.648 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.644+0000 7f1944c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:37.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.669+0000 7f193dd9b640 1 -- 172.21.15.63:0/2595387038 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1938001b70 con 0x7f19401430c0 2024-03-25T18:19:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.669+0000 7f193dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.669+0000 7f193dd9b640 1 -- 172.21.15.63:0/2595387038 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f1938001f80 con 0x7f19401430c0 2024-03-25T18:19:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.669+0000 7f193dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.669+0000 7f193dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.669+0000 7f193dd9b640 10 monclient: dump: 2024-03-25T18:19:37.671 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:37.671 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:37.671 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:37.671 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:37.671 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:37.671 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:37.671 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:37.672 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:37.672 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:37.672 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:37.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.669+0000 7f1944c0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:37.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.669+0000 7f1944c0f640 4 set_mon_vals no callback set 2024-03-25T18:19:37.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.670+0000 7f1944c0f640 10 monclient: shutdown 2024-03-25T18:19:37.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.670+0000 7f1944c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:37.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.670+0000 7f1944c0f640 1 -- 172.21.15.63:0/2595387038 >> v1:172.21.15.63:6789/0 conn(0x7f19401430c0 legacy=0x7f194014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:37.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.670+0000 7f1944c0f640 1 -- 172.21.15.63:0/2595387038 shutdown_connections 2024-03-25T18:19:37.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.670+0000 7f1944c0f640 1 -- 172.21.15.63:0/2595387038 >> 172.21.15.63:0/2595387038 conn(0x7f19400b7ea0 msgr2=0x7f19400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:37.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.670+0000 7f1944c0f640 1 -- 172.21.15.63:0/2595387038 shutdown_connections 2024-03-25T18:19:37.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.670+0000 7f1944c0f640 1 -- 172.21.15.63:0/2595387038 wait complete. 2024-03-25T18:19:37.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.671+0000 7f1944c0f640 10 monclient: build_initial_monmap 2024-03-25T18:19:37.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.671+0000 7f1944c0f640 10 monclient: monmap: 2024-03-25T18:19:37.675 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:19:37.675 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:37.675 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:19:37.675 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:19:37.675 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:37.675 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:19:37.675 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:19:37.676 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:19:37.676 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:19:37.676 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:37.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.672+0000 7f1944c0f640 1 librados: starting msgr at 2024-03-25T18:19:37.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.672+0000 7f1944c0f640 1 librados: starting objecter 2024-03-25T18:19:37.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.672+0000 7f1944c0f640 1 Processor -- start 2024-03-25T18:19:37.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.672+0000 7f1944c0f640 1 -- start start 2024-03-25T18:19:37.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.672+0000 7f1944c0f640 1 librados: setting wanted keys 2024-03-25T18:19:37.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.672+0000 7f1944c0f640 1 librados: calling monclient init 2024-03-25T18:19:37.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.672+0000 7f1944c0f640 10 monclient: init 2024-03-25T18:19:37.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.673+0000 7f1944c0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:37.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.673+0000 7f1944c0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:37.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.673+0000 7f1944c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f19400a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:19:37.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.673+0000 7f1944c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f194013f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:19:37.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.673+0000 7f1944c0f640 10 monclient(hunting): picked mon.noname-a con 0x7f19401430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:19:37.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.673+0000 7f1944c0f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1940136e20 con 0x7f19401430c0 2024-03-25T18:19:37.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.673+0000 7f1944c0f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19400b7940 con 0x7f19400a98f0 2024-03-25T18:19:37.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.673+0000 7f1944c0f640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19401351a0 con 0x7f194013f710 2024-03-25T18:19:37.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.673+0000 7f1944c0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:37.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.673+0000 7f1944c0f640 10 monclient(hunting): authenticate will time out at 7871.859810s 2024-03-25T18:19:37.679 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.678+0000 7f193e59c640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f19400a98f0 0x7f19401daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:55468/0 (socket says 172.21.15.63:55468) 2024-03-25T18:19:37.679 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.678+0000 7f193e59c640 1 -- 172.21.15.63:0/1722253930 learned_addr learned my addr 172.21.15.63:0/1722253930 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:19:37.679 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.678+0000 7f1936ffd640 10 client.?.objecter ms_handle_connect 0x7f19400a98f0 2024-03-25T18:19:37.679 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.678+0000 7f1936ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:37.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.684+0000 7f1936ffd640 10 client.?.objecter ms_handle_connect 0x7f194013f710 2024-03-25T18:19:37.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.684+0000 7f1936ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:37.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.687+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484415291 0 0) 0x7f19400b7940 con 0x7f19400a98f0 2024-03-25T18:19:37.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.687+0000 7f1936ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:37.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.687+0000 7f1936ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:37.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.687+0000 7f1936ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:37.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.687+0000 7f1936ffd640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:19:37.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.687+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1920004eb0 con 0x7f19400a98f0 2024-03-25T18:19:37.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.688+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934612344 0 0) 0x7f19401351a0 con 0x7f194013f710 2024-03-25T18:19:37.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.688+0000 7f1936ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:37.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.688+0000 7f1936ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:37.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.688+0000 7f1936ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:37.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.688+0000 7f1936ffd640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:19:37.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.688+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19400b7940 con 0x7f194013f710 2024-03-25T18:19:37.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.696+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 626942296 0 0) 0x7f1920004eb0 con 0x7f19400a98f0 2024-03-25T18:19:37.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.696+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19401351a0 con 0x7f19400a98f0 2024-03-25T18:19:37.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.696+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f19300019b0 con 0x7f19400a98f0 2024-03-25T18:19:37.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.696+0000 7f1936ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:37.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.696+0000 7f1936ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:37.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.696+0000 7f1936ffd640 10 monclient(hunting): dump: 2024-03-25T18:19:37.699 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:37.699 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:37.699 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:37.699 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:37.699 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:37.699 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:37.699 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:37.699 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:37.699 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:37.699 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:37.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.696+0000 7f1936ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:37.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.696+0000 7f1936ffd640 10 client.?.objecter ms_handle_connect 0x7f19401430c0 2024-03-25T18:19:37.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.696+0000 7f1936ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:37.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2254351749 0 0) 0x7f19401351a0 con 0x7f19400a98f0 2024-03-25T18:19:37.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 >> v1:172.21.15.63:6790/0 conn(0x7f194013f710 legacy=0x7f19401db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:37.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 >> v1:172.21.15.63:6789/0 conn(0x7f19401430c0 legacy=0x7f19401e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:37.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1936ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:37.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1936ffd640 1 monclient: found mon.b 2024-03-25T18:19:37.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1936ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:37.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1936ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:37.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1940135330 con 0x7f19400a98f0 2024-03-25T18:19:37.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1936ffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:37.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1936ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:37.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1936ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:37.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1944c0f640 5 monclient: authenticate success, global_id 4552 2024-03-25T18:19:37.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1944c0f640 10 monclient: _renew_subs 2024-03-25T18:19:37.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1944c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:37.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1944c0f640 1 -- 172.21.15.63:0/1722253930 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1940135560 con 0x7f19400a98f0 2024-03-25T18:19:37.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1944c0f640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:37.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1944c0f640 10 monclient: _renew_subs 2024-03-25T18:19:37.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1944c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:37.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1944c0f640 1 -- 172.21.15.63:0/1722253930 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f194013b490 con 0x7f19400a98f0 2024-03-25T18:19:37.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.707+0000 7f1944c0f640 1 librados: init done 2024-03-25T18:19:37.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.708+0000 7f1944c0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:37.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.708+0000 7f1944c0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:37.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.708+0000 7f1944c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:37.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.708+0000 7f1944c0f640 1 -- 172.21.15.63:0/1722253930 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19400b7730 con 0x7f19400a98f0 2024-03-25T18:19:37.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.983+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19300047e0 con 0x7f19400a98f0 2024-03-25T18:19:37.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.983+0000 7f1936ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:37.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.983+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f1930004b30 con 0x7f19400a98f0 2024-03-25T18:19:37.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.983+0000 7f1936ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:37.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.983+0000 7f1936ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:37.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.983+0000 7f1936ffd640 10 monclient: dump: 2024-03-25T18:19:37.984 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:37.985 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:37.985 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:37.985 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:37.985 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:37.985 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:37.985 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:37.985 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:37.985 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:37.985 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:37.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.983+0000 7f193d59a640 4 set_mon_vals no callback set 2024-03-25T18:19:37.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.984+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f1930004d10 con 0x7f19400a98f0 2024-03-25T18:19:37.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.984+0000 7f1937fff640 10 monclient: get_auth_request con 0x7f192006dcc0 auth_method 0 2024-03-25T18:19:37.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.985+0000 7f1936ffd640 10 client.4552.objecter ms_handle_connect 0x7f192006dcc0 2024-03-25T18:19:37.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.988+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f1930054950 con 0x7f19400a98f0 2024-03-25T18:19:37.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.988+0000 7f1936ffd640 10 client.4552.objecter ms_dispatch 0x7f1940000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:37.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.988+0000 7f1936ffd640 3 client.4552.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:37.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.988+0000 7f1936ffd640 3 client.4552.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:37.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.988+0000 7f1936ffd640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-25T18:19:37.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.989+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1930050290 con 0x7f19400a98f0 2024-03-25T18:19:37.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.989+0000 7f1936ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:37.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:37.989+0000 7f1936ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:38.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.250+0000 7f1944c0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:38.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.250+0000 7f1944c0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:38.252 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.250+0000 7f1944c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:38.252 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.250+0000 7f1944c0f640 1 -- 172.21.15.63:0/1722253930 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19400b7ad0 con 0x7f19400a98f0 2024-03-25T18:19:38.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.268+0000 7f1936ffd640 1 -- 172.21.15.63:0/1722253930 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f1930052390 con 0x7f19400a98f0 2024-03-25T18:19:38.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.268+0000 7f1936ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:38.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.268+0000 7f1936ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:38.269 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:19:38.269 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:38.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.270+0000 7f1944c0f640 10 librados: watch_flush enter 2024-03-25T18:19:38.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.270+0000 7f1944c0f640 10 librados: watch_flush exit 2024-03-25T18:19:38.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.270+0000 7f1944c0f640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-25T18:19:38.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.270+0000 7f1944c0f640 10 client.4552.objecter successfully canceled tick 2024-03-25T18:19:38.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.271+0000 7f1944c0f640 1 -- 172.21.15.63:0/1722253930 >> v1:172.21.15.81:6800/3640340873 conn(0x7f192006dcc0 legacy=0x7f1920070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:38.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.271+0000 7f1944c0f640 10 monclient: shutdown 2024-03-25T18:19:38.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.271+0000 7f1944c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:38.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.271+0000 7f1944c0f640 1 -- 172.21.15.63:0/1722253930 >> v1:172.21.15.81:6789/0 conn(0x7f19400a98f0 legacy=0x7f19401daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:38.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.271+0000 7f1944c0f640 1 -- 172.21.15.63:0/1722253930 shutdown_connections 2024-03-25T18:19:38.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.271+0000 7f1944c0f640 1 -- 172.21.15.63:0/1722253930 >> 172.21.15.63:0/1722253930 conn(0x7f19400b7ea0 msgr2=0x7f1940074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:38.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.271+0000 7f1944c0f640 1 -- 172.21.15.63:0/1722253930 shutdown_connections 2024-03-25T18:19:38.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.271+0000 7f1944c0f640 1 -- 172.21.15.63:0/1722253930 wait complete. 2024-03-25T18:19:38.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:38.271+0000 7f1944c0f640 1 librados: shutdown 2024-03-25T18:19:38.283 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:39.339 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:21.031 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:39.345 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:21.037 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:39.370 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:21.063 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:39.398 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:07:21.058 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:39.408 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:07:21.069 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:39.413 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:21.105 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:39.421 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:21.113 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:39.439 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:07:21.099 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:39.455 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:21.147 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:45.089 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:45.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.199+0000 7f35567ac640 10 monclient: get_monmap_and_config 2024-03-25T18:19:45.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.199+0000 7f35567ac640 10 monclient: build_initial_monmap 2024-03-25T18:19:45.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.199+0000 7f35567ac640 10 monclient: monmap: 2024-03-25T18:19:45.200 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:19:45.200 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:45.201 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:19:45.200654+0000 2024-03-25T18:19:45.201 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:19:45.200654+0000 2024-03-25T18:19:45.201 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:45.201 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:45.201 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:19:45.201 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:19:45.201 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:19:45.201 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:45.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.200+0000 7f35567ac640 1 Processor -- start 2024-03-25T18:19:45.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.200+0000 7f35567ac640 1 -- start start 2024-03-25T18:19:45.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.200+0000 7f35567ac640 10 monclient: init 2024-03-25T18:19:45.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.201+0000 7f35567ac640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:45.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.201+0000 7f35567ac640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:45.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.201+0000 7f35567ac640 10 monclient(hunting): picked mon.noname-c con 0x7f35500a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:19:45.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.201+0000 7f35567ac640 10 monclient(hunting): picked mon.noname-a con 0x7f355013f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:19:45.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.201+0000 7f35567ac640 10 monclient(hunting): picked mon.noname-b con 0x7f35501430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:19:45.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.201+0000 7f35567ac640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3550142520 con 0x7f355013f710 2024-03-25T18:19:45.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.201+0000 7f35567ac640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35500b28a0 con 0x7f35500a98f0 2024-03-25T18:19:45.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.201+0000 7f35567ac640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35500b2a30 con 0x7f35501430c0 2024-03-25T18:19:45.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.201+0000 7f35567ac640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:45.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.201+0000 7f35567ac640 10 monclient(hunting): authenticate will time out at 7879.387529s 2024-03-25T18:19:45.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.214+0000 7f3554d49640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f35501430c0 0x7f355014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:35588/0 (socket says 172.21.15.63:35588) 2024-03-25T18:19:45.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.214+0000 7f3554d49640 1 -- 172.21.15.63:0/1143912289 learned_addr learned my addr 172.21.15.63:0/1143912289 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:19:45.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.226+0000 7f354effd640 1 -- 172.21.15.63:0/1143912289 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975576386 0 0) 0x7f3550142520 con 0x7f355013f710 2024-03-25T18:19:45.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.226+0000 7f354effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.226+0000 7f354effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.226+0000 7f354effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.226+0000 7f354effd640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:19:45.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.226+0000 7f354effd640 1 -- 172.21.15.63:0/1143912289 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3538003520 con 0x7f355013f710 2024-03-25T18:19:45.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.232+0000 7f354effd640 1 -- 172.21.15.63:0/1143912289 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050144413 0 0) 0x7f35500b28a0 con 0x7f35500a98f0 2024-03-25T18:19:45.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.232+0000 7f354effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.232+0000 7f354effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.232+0000 7f354effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.232+0000 7f354effd640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:19:45.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.232+0000 7f354effd640 1 -- 172.21.15.63:0/1143912289 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3550142520 con 0x7f35500a98f0 2024-03-25T18:19:45.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.233+0000 7f354effd640 1 -- 172.21.15.63:0/1143912289 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117175795 0 0) 0x7f35500b2a30 con 0x7f35501430c0 2024-03-25T18:19:45.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.233+0000 7f354effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.233+0000 7f354effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.233+0000 7f354effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.233+0000 7f354effd640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:19:45.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.233+0000 7f354effd640 1 -- 172.21.15.63:0/1143912289 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35500b28a0 con 0x7f35501430c0 2024-03-25T18:19:45.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f354effd640 1 -- 172.21.15.63:0/1143912289 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1886684927 0 0) 0x7f3538003520 con 0x7f355013f710 2024-03-25T18:19:45.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f354effd640 1 -- 172.21.15.63:0/1143912289 >> v1:172.21.15.63:6790/0 conn(0x7f35501430c0 legacy=0x7f355014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f354effd640 1 -- 172.21.15.63:0/1143912289 >> v1:172.21.15.81:6789/0 conn(0x7f35500a98f0 legacy=0x7f35500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f354effd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:45.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f354effd640 1 monclient: found mon.noname-a 2024-03-25T18:19:45.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f354effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:45.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f354effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:19:45.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f354effd640 1 -- 172.21.15.63:0/1143912289 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35500b44c0 con 0x7f355013f710 2024-03-25T18:19:45.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f354effd640 10 monclient: _finish_auth 0 2024-03-25T18:19:45.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f354effd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:45.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f354effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:45.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f354effd640 1 -- 172.21.15.63:0/1143912289 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3544001690 con 0x7f355013f710 2024-03-25T18:19:45.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f35567ac640 5 monclient: authenticate success, global_id 4527 2024-03-25T18:19:45.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f35567ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:45.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f354effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:45.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f354effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:45.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f354effd640 10 monclient: dump: 2024-03-25T18:19:45.239 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:45.239 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:45.239 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:45.239 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:45.239 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:45.239 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:45.239 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:45.240 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:45.240 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:45.240 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:45.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.237+0000 7f35567ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:45.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.244+0000 7f354effd640 1 -- 172.21.15.63:0/1143912289 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35440017f0 con 0x7f355013f710 2024-03-25T18:19:45.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.244+0000 7f354effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:45.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.244+0000 7f354effd640 1 -- 172.21.15.63:0/1143912289 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3544001c00 con 0x7f355013f710 2024-03-25T18:19:45.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.244+0000 7f35567ac640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:45.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.244+0000 7f35567ac640 4 set_mon_vals no callback set 2024-03-25T18:19:45.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.244+0000 7f354effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:45.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.244+0000 7f354effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:45.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.244+0000 7f354effd640 10 monclient: dump: 2024-03-25T18:19:45.246 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:45.246 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:45.246 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:45.246 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:45.246 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:45.246 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:45.246 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:45.246 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:45.246 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:45.246 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:45.248 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.245+0000 7f35567ac640 10 monclient: shutdown 2024-03-25T18:19:45.248 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.245+0000 7f35567ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:45.248 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.245+0000 7f35567ac640 1 -- 172.21.15.63:0/1143912289 >> v1:172.21.15.63:6789/0 conn(0x7f355013f710 legacy=0x7f35501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.248 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.245+0000 7f35567ac640 1 -- 172.21.15.63:0/1143912289 shutdown_connections 2024-03-25T18:19:45.248 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.245+0000 7f35567ac640 1 -- 172.21.15.63:0/1143912289 >> 172.21.15.63:0/1143912289 conn(0x7f35500b7ea0 msgr2=0x7f35500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:45.248 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.245+0000 7f35567ac640 1 -- 172.21.15.63:0/1143912289 shutdown_connections 2024-03-25T18:19:45.248 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.245+0000 7f35567ac640 1 -- 172.21.15.63:0/1143912289 wait complete. 2024-03-25T18:19:45.248 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.246+0000 7f35567ac640 10 monclient: build_initial_monmap 2024-03-25T18:19:45.248 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.246+0000 7f35567ac640 10 monclient: monmap: 2024-03-25T18:19:45.248 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:19:45.248 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:45.249 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:19:45.249 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:19:45.249 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:45.249 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:19:45.249 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:19:45.249 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:19:45.249 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:19:45.249 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:45.249 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.246+0000 7f35567ac640 1 librados: starting msgr at 2024-03-25T18:19:45.249 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.246+0000 7f35567ac640 1 librados: starting objecter 2024-03-25T18:19:45.249 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.246+0000 7f35567ac640 1 Processor -- start 2024-03-25T18:19:45.249 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.246+0000 7f35567ac640 1 -- start start 2024-03-25T18:19:45.249 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.246+0000 7f35567ac640 1 librados: setting wanted keys 2024-03-25T18:19:45.249 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.246+0000 7f35567ac640 1 librados: calling monclient init 2024-03-25T18:19:45.249 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.246+0000 7f35567ac640 10 monclient: init 2024-03-25T18:19:45.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.247+0000 7f35567ac640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:45.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.247+0000 7f35567ac640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:45.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.247+0000 7f35567ac640 10 monclient(hunting): picked mon.noname-b con 0x7f35500a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:19:45.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.247+0000 7f35567ac640 10 monclient(hunting): picked mon.noname-c con 0x7f355013f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:19:45.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.247+0000 7f35567ac640 10 monclient(hunting): picked mon.noname-a con 0x7f35501430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:19:45.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.247+0000 7f35567ac640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3550137f10 con 0x7f35501430c0 2024-03-25T18:19:45.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.247+0000 7f35567ac640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35501dc070 con 0x7f35500a98f0 2024-03-25T18:19:45.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.247+0000 7f35567ac640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3550137100 con 0x7f355013f710 2024-03-25T18:19:45.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.247+0000 7f35567ac640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:45.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.247+0000 7f35567ac640 10 monclient(hunting): authenticate will time out at 7879.434156s 2024-03-25T18:19:45.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.260+0000 7f3554d49640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f35501430c0 0x7f35501db960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:38456/0 (socket says 172.21.15.63:38456) 2024-03-25T18:19:45.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.260+0000 7f3554d49640 1 -- 172.21.15.63:0/818286673 learned_addr learned my addr 172.21.15.63:0/818286673 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:19:45.264 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.263+0000 7f354cff9640 10 client.?.objecter ms_handle_connect 0x7f35501430c0 2024-03-25T18:19:45.264 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.263+0000 7f354cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:45.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.267+0000 7f354cff9640 10 client.?.objecter ms_handle_connect 0x7f35500a98f0 2024-03-25T18:19:45.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.267+0000 7f354cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:45.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.271+0000 7f354cff9640 10 client.?.objecter ms_handle_connect 0x7f355013f710 2024-03-25T18:19:45.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.271+0000 7f354cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:45.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.276+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005287408 0 0) 0x7f3550137100 con 0x7f355013f710 2024-03-25T18:19:45.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.276+0000 7f354cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.276+0000 7f354cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.276+0000 7f354cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.276+0000 7f354cff9640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:19:45.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.276+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f353c004eb0 con 0x7f355013f710 2024-03-25T18:19:45.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.282+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733216182 0 0) 0x7f35501dc070 con 0x7f35500a98f0 2024-03-25T18:19:45.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.282+0000 7f354cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.282+0000 7f354cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.282+0000 7f354cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.282+0000 7f354cff9640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:19:45.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.282+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3550137100 con 0x7f35500a98f0 2024-03-25T18:19:45.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.290+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 38494254 0 0) 0x7f353c004eb0 con 0x7f355013f710 2024-03-25T18:19:45.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.290+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35501dc070 con 0x7f355013f710 2024-03-25T18:19:45.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.290+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3544001b00 con 0x7f355013f710 2024-03-25T18:19:45.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.290+0000 7f354cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:45.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.290+0000 7f354cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:45.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.290+0000 7f354cff9640 10 monclient(hunting): dump: 2024-03-25T18:19:45.291 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:45.292 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:45.292 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:45.292 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:45.292 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:45.292 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:45.292 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:45.292 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:45.292 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:45.292 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:45.292 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.290+0000 7f354cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:45.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.296+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878286821 0 0) 0x7f3550137f10 con 0x7f35501430c0 2024-03-25T18:19:45.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.296+0000 7f354cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.296+0000 7f354cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.296+0000 7f354cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.296+0000 7f354cff9640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:19:45.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.296+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f353c004eb0 con 0x7f35501430c0 2024-03-25T18:19:45.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.302+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1629927648 0 0) 0x7f3550137100 con 0x7f35500a98f0 2024-03-25T18:19:45.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.303+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3550137f10 con 0x7f35500a98f0 2024-03-25T18:19:45.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.303+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3540001970 con 0x7f35500a98f0 2024-03-25T18:19:45.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.303+0000 7f354cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:45.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.303+0000 7f354cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:45.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.303+0000 7f354cff9640 10 monclient(hunting): dump: 2024-03-25T18:19:45.304 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:45.304 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:45.305 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:45.305 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:45.305 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:45.305 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:45.305 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:45.305 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:45.305 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:45.305 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:45.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2536360231 0 0) 0x7f35501dc070 con 0x7f355013f710 2024-03-25T18:19:45.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 >> v1:172.21.15.81:6789/0 conn(0x7f35500a98f0 legacy=0x7f3550137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 >> v1:172.21.15.63:6789/0 conn(0x7f35501430c0 legacy=0x7f35501db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f354cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:45.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f354cff9640 1 monclient: found mon.c 2024-03-25T18:19:45.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f354cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:45.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f354cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:19:45.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35501373e0 con 0x7f355013f710 2024-03-25T18:19:45.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f354cff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:45.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f354cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:45.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f354cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:45.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f35567ac640 5 monclient: authenticate success, global_id 4535 2024-03-25T18:19:45.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f35567ac640 10 monclient: _renew_subs 2024-03-25T18:19:45.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f35567ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:19:45.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f35567ac640 1 -- 172.21.15.63:0/818286673 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35501358e0 con 0x7f355013f710 2024-03-25T18:19:45.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f35567ac640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:45.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f35567ac640 10 monclient: _renew_subs 2024-03-25T18:19:45.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f35567ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:19:45.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f35567ac640 1 -- 172.21.15.63:0/818286673 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3550136150 con 0x7f355013f710 2024-03-25T18:19:45.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.305+0000 7f35567ac640 1 librados: init done 2024-03-25T18:19:45.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.306+0000 7f35567ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:45.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.306+0000 7f35567ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:45.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.306+0000 7f35567ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:19:45.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.306+0000 7f35567ac640 1 -- 172.21.15.63:0/818286673 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35500627b0 con 0x7f355013f710 2024-03-25T18:19:45.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.617+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35440038a0 con 0x7f355013f710 2024-03-25T18:19:45.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.617+0000 7f354cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:45.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.617+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3544003bb0 con 0x7f355013f710 2024-03-25T18:19:45.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.617+0000 7f354cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:45.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.617+0000 7f354cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:45.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.617+0000 7f354cff9640 10 monclient: dump: 2024-03-25T18:19:45.619 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:45.619 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:45.619 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:45.619 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:45.619 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:45.619 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:45.619 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:45.619 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:45.619 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:45.620 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:45.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.617+0000 7f354e7fc640 4 set_mon_vals no callback set 2024-03-25T18:19:45.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.618+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f3544019b80 con 0x7f355013f710 2024-03-25T18:19:45.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.619+0000 7f354ffff640 10 monclient: get_auth_request con 0x7f353c06dfb0 auth_method 0 2024-03-25T18:19:45.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.619+0000 7f354cff9640 10 client.4535.objecter ms_handle_connect 0x7f353c06dfb0 2024-03-25T18:19:45.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.624+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f3544053650 con 0x7f355013f710 2024-03-25T18:19:45.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.624+0000 7f354cff9640 10 client.4535.objecter ms_dispatch 0x7f3550000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:45.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.624+0000 7f354cff9640 3 client.4535.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:45.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.624+0000 7f354cff9640 3 client.4535.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:45.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.624+0000 7f354cff9640 20 client.4535.objecter dump_active .. 0 homeless 2024-03-25T18:19:45.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.624+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f354404f160 con 0x7f355013f710 2024-03-25T18:19:45.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.624+0000 7f354cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:45.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.624+0000 7f354cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:45.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.877+0000 7f35567ac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:45.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.877+0000 7f35567ac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:45.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.877+0000 7f35567ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:19:45.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:45.877+0000 7f35567ac640 1 -- 172.21.15.63:0/818286673 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f355013e250 con 0x7f355013f710 2024-03-25T18:19:46.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:46.043+0000 7f354cff9640 1 -- 172.21.15.63:0/818286673 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f3544051090 con 0x7f355013f710 2024-03-25T18:19:46.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:46.043+0000 7f354cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:46.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:46.043+0000 7f354cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:46.045 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:19:46.045 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:46.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:46.045+0000 7f35567ac640 10 librados: watch_flush enter 2024-03-25T18:19:46.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:46.045+0000 7f35567ac640 10 librados: watch_flush exit 2024-03-25T18:19:46.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:46.045+0000 7f35567ac640 20 client.4535.objecter shutdown clearing up homeless session... 2024-03-25T18:19:46.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:46.045+0000 7f35567ac640 10 client.4535.objecter successfully canceled tick 2024-03-25T18:19:46.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:46.045+0000 7f35567ac640 1 -- 172.21.15.63:0/818286673 >> v1:172.21.15.81:6800/3640340873 conn(0x7f353c06dfb0 legacy=0x7f353c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:46.045+0000 7f35567ac640 10 monclient: shutdown 2024-03-25T18:19:46.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:46.045+0000 7f35567ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:46.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:46.045+0000 7f35567ac640 1 -- 172.21.15.63:0/818286673 >> v1:172.21.15.63:6790/0 conn(0x7f355013f710 legacy=0x7f35501db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:46.045+0000 7f35567ac640 1 -- 172.21.15.63:0/818286673 shutdown_connections 2024-03-25T18:19:46.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:46.045+0000 7f35567ac640 1 -- 172.21.15.63:0/818286673 >> 172.21.15.63:0/818286673 conn(0x7f35500b7ea0 msgr2=0x7f355014ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:46.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:46.046+0000 7f35567ac640 1 -- 172.21.15.63:0/818286673 shutdown_connections 2024-03-25T18:19:46.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:46.046+0000 7f35567ac640 1 -- 172.21.15.63:0/818286673 wait complete. 2024-03-25T18:19:46.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:46.046+0000 7f35567ac640 1 librados: shutdown 2024-03-25T18:19:46.057 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:52.863 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:52.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.976+0000 7fbc7f6f4640 10 monclient: get_monmap_and_config 2024-03-25T18:19:52.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.976+0000 7fbc7f6f4640 10 monclient: build_initial_monmap 2024-03-25T18:19:52.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.976+0000 7fbc7f6f4640 10 monclient: monmap: 2024-03-25T18:19:52.977 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:19:52.977 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:52.977 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:19:52.977698+0000 2024-03-25T18:19:52.978 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:19:52.977698+0000 2024-03-25T18:19:52.978 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:52.978 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:52.978 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:19:52.978 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:19:52.978 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:19:52.978 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:52.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.977+0000 7fbc7f6f4640 1 Processor -- start 2024-03-25T18:19:52.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.977+0000 7fbc7f6f4640 1 -- start start 2024-03-25T18:19:52.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.977+0000 7fbc7f6f4640 10 monclient: init 2024-03-25T18:19:52.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.978+0000 7fbc7f6f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:52.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.978+0000 7fbc7f6f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:52.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.978+0000 7fbc7f6f4640 10 monclient(hunting): picked mon.noname-c con 0x7fbc780a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:19:52.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.978+0000 7fbc7f6f4640 10 monclient(hunting): picked mon.noname-a con 0x7fbc7813f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:19:52.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.978+0000 7fbc7f6f4640 10 monclient(hunting): picked mon.noname-b con 0x7fbc781430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:19:52.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.978+0000 7fbc7f6f4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc78142520 con 0x7fbc7813f710 2024-03-25T18:19:52.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.978+0000 7fbc7f6f4640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc780b28a0 con 0x7fbc780a98f0 2024-03-25T18:19:52.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.978+0000 7fbc7f6f4640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc780b2a30 con 0x7fbc781430c0 2024-03-25T18:19:52.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.978+0000 7fbc7f6f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:52.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.978+0000 7fbc7f6f4640 10 monclient(hunting): authenticate will time out at 7887.164765s 2024-03-25T18:19:52.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.994+0000 7fbc7cc8f640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fbc7813f710 0x7fbc781429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:37574/0 (socket says 172.21.15.63:37574) 2024-03-25T18:19:52.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:52.994+0000 7fbc7cc8f640 1 -- 172.21.15.63:0/2493094346 learned_addr learned my addr 172.21.15.63:0/2493094346 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:19:53.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.012+0000 7fbc6ffff640 1 -- 172.21.15.63:0/2493094346 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730261998 0 0) 0x7fbc780b2a30 con 0x7fbc781430c0 2024-03-25T18:19:53.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.012+0000 7fbc6ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:53.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.012+0000 7fbc6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:53.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.012+0000 7fbc6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:53.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.012+0000 7fbc6ffff640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:19:53.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.012+0000 7fbc6ffff640 1 -- 172.21.15.63:0/2493094346 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc64003520 con 0x7fbc781430c0 2024-03-25T18:19:53.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.016+0000 7fbc6ffff640 1 -- 172.21.15.63:0/2493094346 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310755922 0 0) 0x7fbc780b28a0 con 0x7fbc780a98f0 2024-03-25T18:19:53.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.016+0000 7fbc6ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:53.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.016+0000 7fbc6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:53.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.016+0000 7fbc6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:53.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.016+0000 7fbc6ffff640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:19:53.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.016+0000 7fbc6ffff640 1 -- 172.21.15.63:0/2493094346 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc64003840 con 0x7fbc780a98f0 2024-03-25T18:19:53.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.021+0000 7fbc6ffff640 1 -- 172.21.15.63:0/2493094346 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2257687332 0 0) 0x7fbc64003520 con 0x7fbc781430c0 2024-03-25T18:19:53.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.021+0000 7fbc6ffff640 1 -- 172.21.15.63:0/2493094346 >> v1:172.21.15.81:6789/0 conn(0x7fbc780a98f0 legacy=0x7fbc780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:53.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.021+0000 7fbc6ffff640 1 -- 172.21.15.63:0/2493094346 >> v1:172.21.15.63:6789/0 conn(0x7fbc7813f710 legacy=0x7fbc781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:53.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.021+0000 7fbc6ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:53.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.021+0000 7fbc6ffff640 1 monclient: found mon.noname-b 2024-03-25T18:19:53.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.021+0000 7fbc6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:53.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.021+0000 7fbc6ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:19:53.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.021+0000 7fbc6ffff640 1 -- 172.21.15.63:0/2493094346 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc780b44c0 con 0x7fbc781430c0 2024-03-25T18:19:53.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.021+0000 7fbc6ffff640 10 monclient: _finish_auth 0 2024-03-25T18:19:53.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.021+0000 7fbc6ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:53.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.022+0000 7fbc6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:53.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.022+0000 7fbc6ffff640 1 -- 172.21.15.63:0/2493094346 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fbc700019f0 con 0x7fbc781430c0 2024-03-25T18:19:53.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.022+0000 7fbc7f6f4640 5 monclient: authenticate success, global_id 4538 2024-03-25T18:19:53.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.022+0000 7fbc7f6f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:53.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.022+0000 7fbc6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:53.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.022+0000 7fbc6ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:53.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.022+0000 7fbc6ffff640 10 monclient: dump: 2024-03-25T18:19:53.024 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:53.024 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:53.024 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:53.024 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:53.024 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:53.024 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:53.024 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:53.024 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:53.024 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:53.025 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:53.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.022+0000 7fbc7f6f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:53.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.031+0000 7fbc6ffff640 1 -- 172.21.15.63:0/2493094346 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc70001b50 con 0x7fbc781430c0 2024-03-25T18:19:53.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.031+0000 7fbc6ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:53.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.031+0000 7fbc6ffff640 1 -- 172.21.15.63:0/2493094346 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fbc70001f60 con 0x7fbc781430c0 2024-03-25T18:19:53.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.031+0000 7fbc6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:53.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.031+0000 7fbc6ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:53.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.031+0000 7fbc6ffff640 10 monclient: dump: 2024-03-25T18:19:53.032 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:53.033 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:53.033 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:53.033 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:53.033 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:53.033 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:53.033 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:53.033 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:53.033 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:53.033 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:53.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.031+0000 7fbc7f6f4640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:53.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.031+0000 7fbc7f6f4640 4 set_mon_vals no callback set 2024-03-25T18:19:53.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.032+0000 7fbc7f6f4640 10 monclient: shutdown 2024-03-25T18:19:53.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.032+0000 7fbc7f6f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:53.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.032+0000 7fbc7f6f4640 1 -- 172.21.15.63:0/2493094346 >> v1:172.21.15.63:6790/0 conn(0x7fbc781430c0 legacy=0x7fbc7814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:53.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.032+0000 7fbc7f6f4640 1 -- 172.21.15.63:0/2493094346 shutdown_connections 2024-03-25T18:19:53.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.032+0000 7fbc7f6f4640 1 -- 172.21.15.63:0/2493094346 >> 172.21.15.63:0/2493094346 conn(0x7fbc780b7ea0 msgr2=0x7fbc780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:53.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.032+0000 7fbc7f6f4640 1 -- 172.21.15.63:0/2493094346 shutdown_connections 2024-03-25T18:19:53.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.032+0000 7fbc7f6f4640 1 -- 172.21.15.63:0/2493094346 wait complete. 2024-03-25T18:19:53.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.033+0000 7fbc7f6f4640 10 monclient: build_initial_monmap 2024-03-25T18:19:53.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.033+0000 7fbc7f6f4640 10 monclient: monmap: 2024-03-25T18:19:53.036 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:19:53.036 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:53.036 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:19:53.036 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:19:53.036 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:53.036 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:19:53.036 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:19:53.036 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:19:53.037 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:19:53.037 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:53.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.033+0000 7fbc7f6f4640 1 librados: starting msgr at 2024-03-25T18:19:53.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.033+0000 7fbc7f6f4640 1 librados: starting objecter 2024-03-25T18:19:53.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.033+0000 7fbc7f6f4640 1 Processor -- start 2024-03-25T18:19:53.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.033+0000 7fbc7f6f4640 1 -- start start 2024-03-25T18:19:53.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.033+0000 7fbc7f6f4640 1 librados: setting wanted keys 2024-03-25T18:19:53.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.033+0000 7fbc7f6f4640 1 librados: calling monclient init 2024-03-25T18:19:53.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.033+0000 7fbc7f6f4640 10 monclient: init 2024-03-25T18:19:53.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.034+0000 7fbc7f6f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:53.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.034+0000 7fbc7f6f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:53.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.034+0000 7fbc7f6f4640 10 monclient(hunting): picked mon.noname-c con 0x7fbc780a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:19:53.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.034+0000 7fbc7f6f4640 10 monclient(hunting): picked mon.noname-b con 0x7fbc7813f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:19:53.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.034+0000 7fbc7f6f4640 10 monclient(hunting): picked mon.noname-a con 0x7fbc781430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:19:53.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.034+0000 7fbc7f6f4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc78137ab0 con 0x7fbc781430c0 2024-03-25T18:19:53.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.034+0000 7fbc7f6f4640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc78138bd0 con 0x7fbc7813f710 2024-03-25T18:19:53.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.034+0000 7fbc7f6f4640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc78138e40 con 0x7fbc780a98f0 2024-03-25T18:19:53.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.034+0000 7fbc7f6f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:53.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.034+0000 7fbc7f6f4640 10 monclient(hunting): authenticate will time out at 7887.220929s 2024-03-25T18:19:53.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.042+0000 7fbc7cc8f640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fbc7813f710 0x7fbc781df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:41280/0 (socket says 172.21.15.63:41280) 2024-03-25T18:19:53.043 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.042+0000 7fbc7cc8f640 1 -- 172.21.15.63:0/2181612386 learned_addr learned my addr 172.21.15.63:0/2181612386 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:19:53.043 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.042+0000 7fbc6dffb640 10 client.?.objecter ms_handle_connect 0x7fbc7813f710 2024-03-25T18:19:53.043 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.042+0000 7fbc6dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:53.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.050+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094111462 0 0) 0x7fbc78138bd0 con 0x7fbc7813f710 2024-03-25T18:19:53.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.050+0000 7fbc6dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:53.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.050+0000 7fbc6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:53.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.050+0000 7fbc6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:53.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.050+0000 7fbc6dffb640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:19:53.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.050+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc5c004eb0 con 0x7fbc7813f710 2024-03-25T18:19:53.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.052+0000 7fbc6dffb640 10 client.?.objecter ms_handle_connect 0x7fbc780a98f0 2024-03-25T18:19:53.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.052+0000 7fbc6dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:53.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.052+0000 7fbc6dffb640 10 client.?.objecter ms_handle_connect 0x7fbc781430c0 2024-03-25T18:19:53.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.052+0000 7fbc6dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:53.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.060+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3474872120 0 0) 0x7fbc5c004eb0 con 0x7fbc7813f710 2024-03-25T18:19:53.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.060+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc78138bd0 con 0x7fbc7813f710 2024-03-25T18:19:53.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.060+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fbc680018b0 con 0x7fbc7813f710 2024-03-25T18:19:53.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.060+0000 7fbc6dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:53.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.060+0000 7fbc6dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:53.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.060+0000 7fbc6dffb640 10 monclient(hunting): dump: 2024-03-25T18:19:53.061 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:53.061 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:53.062 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:53.062 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:53.062 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:53.062 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:53.062 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:53.062 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:53.062 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:53.062 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:53.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.060+0000 7fbc6dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:53.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.063+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049518456 0 0) 0x7fbc78137ab0 con 0x7fbc781430c0 2024-03-25T18:19:53.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.063+0000 7fbc6dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:53.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.063+0000 7fbc6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:53.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.063+0000 7fbc6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:53.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.063+0000 7fbc6dffb640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:19:53.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.063+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc5c004eb0 con 0x7fbc781430c0 2024-03-25T18:19:53.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.064+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 119573615 0 0) 0x7fbc78138bd0 con 0x7fbc7813f710 2024-03-25T18:19:53.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.064+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 >> v1:172.21.15.63:6790/0 conn(0x7fbc780a98f0 legacy=0x7fbc78137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:53.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.064+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 >> v1:172.21.15.63:6789/0 conn(0x7fbc781430c0 legacy=0x7fbc781e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:53.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.064+0000 7fbc6dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:53.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.064+0000 7fbc6dffb640 1 monclient: found mon.b 2024-03-25T18:19:53.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.064+0000 7fbc6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:53.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.064+0000 7fbc6dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:53.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.064+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc78139120 con 0x7fbc7813f710 2024-03-25T18:19:53.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.064+0000 7fbc6dffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:53.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.064+0000 7fbc6dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:53.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.064+0000 7fbc6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:53.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.064+0000 7fbc7f6f4640 5 monclient: authenticate success, global_id 4564 2024-03-25T18:19:53.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.065+0000 7fbc7f6f4640 10 monclient: _renew_subs 2024-03-25T18:19:53.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.065+0000 7fbc7f6f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:53.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.065+0000 7fbc7f6f4640 1 -- 172.21.15.63:0/2181612386 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc78139400 con 0x7fbc7813f710 2024-03-25T18:19:53.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.065+0000 7fbc7f6f4640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:53.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.065+0000 7fbc7f6f4640 10 monclient: _renew_subs 2024-03-25T18:19:53.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.065+0000 7fbc7f6f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:53.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.065+0000 7fbc7f6f4640 1 -- 172.21.15.63:0/2181612386 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc780b7880 con 0x7fbc7813f710 2024-03-25T18:19:53.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.065+0000 7fbc7f6f4640 1 librados: init done 2024-03-25T18:19:53.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.066+0000 7fbc7f6f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:53.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.066+0000 7fbc7f6f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:53.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.066+0000 7fbc7f6f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:53.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.066+0000 7fbc7f6f4640 1 -- 172.21.15.63:0/2181612386 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc78137810 con 0x7fbc7813f710 2024-03-25T18:19:53.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.345+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc680047e0 con 0x7fbc7813f710 2024-03-25T18:19:53.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.345+0000 7fbc6dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:53.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.345+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fbc68004c30 con 0x7fbc7813f710 2024-03-25T18:19:53.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.345+0000 7fbc6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:53.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.345+0000 7fbc6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:53.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.345+0000 7fbc6dffb640 10 monclient: dump: 2024-03-25T18:19:53.347 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:19:53.347 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:19:53.347 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:53.347 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:19:53.347 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:53.347 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:19:53.347 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:19:53.347 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:19:53.347 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:19:53.348 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:19:53.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.345+0000 7fbc6f7fe640 4 set_mon_vals no callback set 2024-03-25T18:19:53.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.346+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fbc68004e50 con 0x7fbc7813f710 2024-03-25T18:19:53.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.347+0000 7fbc7d490640 10 monclient: get_auth_request con 0x7fbc5c06dd90 auth_method 0 2024-03-25T18:19:53.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.347+0000 7fbc6dffb640 10 client.4564.objecter ms_handle_connect 0x7fbc5c06dd90 2024-03-25T18:19:53.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.352+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fbc68054800 con 0x7fbc7813f710 2024-03-25T18:19:53.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.352+0000 7fbc6dffb640 10 client.4564.objecter ms_dispatch 0x7fbc78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:53.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.352+0000 7fbc6dffb640 3 client.4564.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:53.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.352+0000 7fbc6dffb640 3 client.4564.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:53.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.352+0000 7fbc6dffb640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-25T18:19:53.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.352+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc68050310 con 0x7fbc7813f710 2024-03-25T18:19:53.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.352+0000 7fbc6dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:53.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.352+0000 7fbc6dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:53.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.572+0000 7fbc7f6f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:53.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.572+0000 7fbc7f6f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:53.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.572+0000 7fbc7f6f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:19:53.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.572+0000 7fbc7f6f4640 1 -- 172.21.15.63:0/2181612386 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc78137c60 con 0x7fbc7813f710 2024-03-25T18:19:53.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.607+0000 7fbc6dffb640 1 -- 172.21.15.63:0/2181612386 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fbc78137c60 con 0x7fbc7813f710 2024-03-25T18:19:53.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.607+0000 7fbc6dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:53.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.607+0000 7fbc6dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:53.608 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:19:53.608 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:53.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.609+0000 7fbc7f6f4640 10 librados: watch_flush enter 2024-03-25T18:19:53.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.609+0000 7fbc7f6f4640 10 librados: watch_flush exit 2024-03-25T18:19:53.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.609+0000 7fbc7f6f4640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-25T18:19:53.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.609+0000 7fbc7f6f4640 10 client.4564.objecter successfully canceled tick 2024-03-25T18:19:53.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.609+0000 7fbc7f6f4640 1 -- 172.21.15.63:0/2181612386 >> v1:172.21.15.81:6800/3640340873 conn(0x7fbc5c06dd90 legacy=0x7fbc5c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:53.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.609+0000 7fbc7f6f4640 10 monclient: shutdown 2024-03-25T18:19:53.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.609+0000 7fbc7f6f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:53.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.609+0000 7fbc7f6f4640 1 -- 172.21.15.63:0/2181612386 >> v1:172.21.15.81:6789/0 conn(0x7fbc7813f710 legacy=0x7fbc781df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:53.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.609+0000 7fbc7f6f4640 1 -- 172.21.15.63:0/2181612386 shutdown_connections 2024-03-25T18:19:53.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.609+0000 7fbc7f6f4640 1 -- 172.21.15.63:0/2181612386 >> 172.21.15.63:0/2181612386 conn(0x7fbc780b7ea0 msgr2=0x7fbc78074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:53.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.609+0000 7fbc7f6f4640 1 -- 172.21.15.63:0/2181612386 shutdown_connections 2024-03-25T18:19:53.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.609+0000 7fbc7f6f4640 1 -- 172.21.15.63:0/2181612386 wait complete. 2024-03-25T18:19:53.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:19:53.609+0000 7fbc7f6f4640 1 librados: shutdown 2024-03-25T18:19:53.621 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:59.415 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:41.107 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:59.418 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:41.110 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:59.457 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:41.149 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:59.480 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:41.173 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:59.484 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:41.177 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:59.494 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:07:41.154 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:59.498 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:07:41.158 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:59.514 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:41.206 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:59.520 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:07:41.180 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:00.427 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:00.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.526+0000 7f93d8723640 10 monclient: get_monmap_and_config 2024-03-25T18:20:00.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.526+0000 7f93d8723640 10 monclient: build_initial_monmap 2024-03-25T18:20:00.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.526+0000 7f93d8723640 10 monclient: monmap: 2024-03-25T18:20:00.526 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:20:00.526 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:00.526 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:20:00.526425+0000 2024-03-25T18:20:00.527 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:20:00.526425+0000 2024-03-25T18:20:00.527 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:00.527 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:00.527 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:20:00.527 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:20:00.527 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:20:00.527 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:00.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.527+0000 7f93d8723640 1 Processor -- start 2024-03-25T18:20:00.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.527+0000 7f93d8723640 1 -- start start 2024-03-25T18:20:00.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.527+0000 7f93d8723640 10 monclient: init 2024-03-25T18:20:00.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.528+0000 7f93d8723640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:00.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.528+0000 7f93d8723640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:00.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.528+0000 7f93d8723640 10 monclient(hunting): picked mon.noname-a con 0x7f93d013fc60 addr v1:172.21.15.63:6789/0 2024-03-25T18:20:00.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.528+0000 7f93d8723640 10 monclient(hunting): picked mon.noname-b con 0x7f93d0070570 addr v1:172.21.15.63:6790/0 2024-03-25T18:20:00.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.528+0000 7f93d8723640 10 monclient(hunting): picked mon.noname-c con 0x7f93d00727a0 addr v1:172.21.15.81:6789/0 2024-03-25T18:20:00.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.528+0000 7f93d8723640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d013c9f0 con 0x7f93d013fc60 2024-03-25T18:20:00.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.528+0000 7f93d8723640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d006fac0 con 0x7f93d00727a0 2024-03-25T18:20:00.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.528+0000 7f93d8723640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d00715d0 con 0x7f93d0070570 2024-03-25T18:20:00.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.528+0000 7f93d8723640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:00.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.528+0000 7f93d8723640 10 monclient(hunting): authenticate will time out at 7894.713822s 2024-03-25T18:20:00.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.534+0000 7f93d6cc0640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f93d00727a0 0x7f93d014a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:55836/0 (socket says 172.21.15.63:55836) 2024-03-25T18:20:00.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.534+0000 7f93d6cc0640 1 -- 172.21.15.63:0/1046735090 learned_addr learned my addr 172.21.15.63:0/1046735090 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:20:00.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.548+0000 7f93d54bd640 1 -- 172.21.15.63:0/1046735090 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088391021 0 0) 0x7f93d006fac0 con 0x7f93d00727a0 2024-03-25T18:20:00.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.548+0000 7f93d54bd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:00.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.548+0000 7f93d54bd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:00.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.548+0000 7f93d54bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:00.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.548+0000 7f93d54bd640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:20:00.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.548+0000 7f93d54bd640 1 -- 172.21.15.63:0/1046735090 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93bc003520 con 0x7f93d00727a0 2024-03-25T18:20:00.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.554+0000 7f93d54bd640 1 -- 172.21.15.63:0/1046735090 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772536512 0 0) 0x7f93d013c9f0 con 0x7f93d013fc60 2024-03-25T18:20:00.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.554+0000 7f93d54bd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:00.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.554+0000 7f93d54bd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:00.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.554+0000 7f93d54bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:00.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.554+0000 7f93d54bd640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:20:00.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.554+0000 7f93d54bd640 1 -- 172.21.15.63:0/1046735090 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93d006fac0 con 0x7f93d013fc60 2024-03-25T18:20:00.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d54bd640 1 -- 172.21.15.63:0/1046735090 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2630507784 0 0) 0x7f93bc003520 con 0x7f93d00727a0 2024-03-25T18:20:00.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d54bd640 1 -- 172.21.15.63:0/1046735090 >> v1:172.21.15.63:6790/0 conn(0x7f93d0070570 legacy=0x7f93d0072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:00.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d54bd640 1 -- 172.21.15.63:0/1046735090 >> v1:172.21.15.63:6789/0 conn(0x7f93d013fc60 legacy=0x7f93d006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:00.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d54bd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:00.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d54bd640 1 monclient: found mon.noname-c 2024-03-25T18:20:00.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d54bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:00.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d54bd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:20:00.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d54bd640 1 -- 172.21.15.63:0/1046735090 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d00718b0 con 0x7f93d00727a0 2024-03-25T18:20:00.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d54bd640 10 monclient: _finish_auth 0 2024-03-25T18:20:00.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d54bd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:00.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d54bd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:00.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d54bd640 1 -- 172.21.15.63:0/1046735090 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f93c8001a10 con 0x7f93d00727a0 2024-03-25T18:20:00.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d54bd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:00.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d54bd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:00.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d54bd640 10 monclient: dump: 2024-03-25T18:20:00.559 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:00.559 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:00.559 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:00.559 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:00.559 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:00.559 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:00.559 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:00.559 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:00.559 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:00.559 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:00.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d8723640 5 monclient: authenticate success, global_id 4567 2024-03-25T18:20:00.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.557+0000 7f93d8723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:00.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.563+0000 7f93d54bd640 1 -- 172.21.15.63:0/1046735090 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93c8001b70 con 0x7f93d00727a0 2024-03-25T18:20:00.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.563+0000 7f93d54bd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:00.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.563+0000 7f93d54bd640 1 -- 172.21.15.63:0/1046735090 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f93c8001f80 con 0x7f93d00727a0 2024-03-25T18:20:00.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.563+0000 7f93d54bd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:00.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.563+0000 7f93d54bd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:00.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.563+0000 7f93d54bd640 10 monclient: dump: 2024-03-25T18:20:00.564 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:00.564 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:00.564 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:00.564 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:00.564 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:00.564 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:00.564 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:00.564 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:00.564 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:00.564 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:00.564 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.563+0000 7f93d8723640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:00.564 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.563+0000 7f93d8723640 4 set_mon_vals no callback set 2024-03-25T18:20:00.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.564+0000 7f93d8723640 10 monclient: shutdown 2024-03-25T18:20:00.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.564+0000 7f93d8723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:00.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.564+0000 7f93d8723640 1 -- 172.21.15.63:0/1046735090 >> v1:172.21.15.81:6789/0 conn(0x7f93d00727a0 legacy=0x7f93d014a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:00.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.564+0000 7f93d8723640 1 -- 172.21.15.63:0/1046735090 shutdown_connections 2024-03-25T18:20:00.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.564+0000 7f93d8723640 1 -- 172.21.15.63:0/1046735090 >> 172.21.15.63:0/1046735090 conn(0x7f93d00b4a80 msgr2=0x7f93d00b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:00.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.564+0000 7f93d8723640 1 -- 172.21.15.63:0/1046735090 shutdown_connections 2024-03-25T18:20:00.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.564+0000 7f93d8723640 1 -- 172.21.15.63:0/1046735090 wait complete. 2024-03-25T18:20:00.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.565+0000 7f93d8723640 10 monclient: build_initial_monmap 2024-03-25T18:20:00.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.565+0000 7f93d8723640 10 monclient: monmap: 2024-03-25T18:20:00.567 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:20:00.567 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:00.567 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:20:00.567 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:20:00.567 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:00.567 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:20:00.567 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:20:00.567 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:20:00.567 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:20:00.568 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:00.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.565+0000 7f93d8723640 1 librados: starting msgr at 2024-03-25T18:20:00.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.565+0000 7f93d8723640 1 librados: starting objecter 2024-03-25T18:20:00.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.565+0000 7f93d8723640 1 Processor -- start 2024-03-25T18:20:00.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.565+0000 7f93d8723640 1 -- start start 2024-03-25T18:20:00.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.565+0000 7f93d8723640 1 librados: setting wanted keys 2024-03-25T18:20:00.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.565+0000 7f93d8723640 1 librados: calling monclient init 2024-03-25T18:20:00.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.565+0000 7f93d8723640 10 monclient: init 2024-03-25T18:20:00.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.566+0000 7f93d8723640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:00.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.566+0000 7f93d8723640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:00.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.566+0000 7f93d8723640 10 monclient(hunting): picked mon.noname-a con 0x7f93d0070570 addr v1:172.21.15.63:6789/0 2024-03-25T18:20:00.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.566+0000 7f93d8723640 10 monclient(hunting): picked mon.noname-c con 0x7f93d00727a0 addr v1:172.21.15.63:6790/0 2024-03-25T18:20:00.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.566+0000 7f93d8723640 10 monclient(hunting): picked mon.noname-b con 0x7f93d013fc60 addr v1:172.21.15.81:6789/0 2024-03-25T18:20:00.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.566+0000 7f93d8723640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d0141d10 con 0x7f93d0070570 2024-03-25T18:20:00.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.566+0000 7f93d8723640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d01417f0 con 0x7f93d013fc60 2024-03-25T18:20:00.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.566+0000 7f93d8723640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d0140740 con 0x7f93d00727a0 2024-03-25T18:20:00.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.566+0000 7f93d8723640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:00.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.566+0000 7f93d8723640 10 monclient(hunting): authenticate will time out at 7894.751691s 2024-03-25T18:20:00.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.569+0000 7f93d6cc0640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f93d013fc60 0x7f93d01d76b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:55880/0 (socket says 172.21.15.63:55880) 2024-03-25T18:20:00.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.569+0000 7f93d6cc0640 1 -- 172.21.15.63:0/3051759945 learned_addr learned my addr 172.21.15.63:0/3051759945 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:20:00.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.569+0000 7f93c6ffd640 10 client.?.objecter ms_handle_connect 0x7f93d013fc60 2024-03-25T18:20:00.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.569+0000 7f93c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:00.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.576+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533018704 0 0) 0x7f93d01417f0 con 0x7f93d013fc60 2024-03-25T18:20:00.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.576+0000 7f93c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:00.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.576+0000 7f93c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:00.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.576+0000 7f93c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:00.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.576+0000 7f93c6ffd640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:20:00.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.576+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93b4004eb0 con 0x7f93d013fc60 2024-03-25T18:20:00.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.577+0000 7f93c6ffd640 10 client.?.objecter ms_handle_connect 0x7f93d00727a0 2024-03-25T18:20:00.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.577+0000 7f93c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:00.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.583+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4072242097 0 0) 0x7f93b4004eb0 con 0x7f93d013fc60 2024-03-25T18:20:00.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.583+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93d01417f0 con 0x7f93d013fc60 2024-03-25T18:20:00.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.583+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f93c8001e80 con 0x7f93d013fc60 2024-03-25T18:20:00.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.583+0000 7f93c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:00.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.583+0000 7f93c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:00.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.583+0000 7f93c6ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:00.584 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:00.584 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:00.584 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:00.584 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:00.584 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:00.584 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:00.584 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:00.584 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:00.584 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:00.584 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:00.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.583+0000 7f93c6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:00.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.585+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646037912 0 0) 0x7f93d0140740 con 0x7f93d00727a0 2024-03-25T18:20:00.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.585+0000 7f93c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:00.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.585+0000 7f93c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:00.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.585+0000 7f93c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:00.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.585+0000 7f93c6ffd640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:20:00.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.585+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93b4004eb0 con 0x7f93d00727a0 2024-03-25T18:20:00.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.586+0000 7f93c6ffd640 10 client.?.objecter ms_handle_connect 0x7f93d0070570 2024-03-25T18:20:00.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.586+0000 7f93c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:00.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2521662121 0 0) 0x7f93d01417f0 con 0x7f93d013fc60 2024-03-25T18:20:00.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 >> v1:172.21.15.63:6790/0 conn(0x7f93d00727a0 legacy=0x7f93d0136d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:00.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 >> v1:172.21.15.63:6789/0 conn(0x7f93d0070570 legacy=0x7f93d0140f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:00.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:00.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93c6ffd640 1 monclient: found mon.b 2024-03-25T18:20:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93c6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d0137460 con 0x7f93d013fc60 2024-03-25T18:20:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93c6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:20:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93d8723640 5 monclient: authenticate success, global_id 4570 2024-03-25T18:20:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93d8723640 10 monclient: _renew_subs 2024-03-25T18:20:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93d8723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93d8723640 1 -- 172.21.15.63:0/3051759945 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d00adaf0 con 0x7f93d013fc60 2024-03-25T18:20:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93d8723640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93d8723640 10 monclient: _renew_subs 2024-03-25T18:20:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93d8723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93d8723640 1 -- 172.21.15.63:0/3051759945 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d01d7fa0 con 0x7f93d013fc60 2024-03-25T18:20:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.591+0000 7f93d8723640 1 librados: init done 2024-03-25T18:20:00.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.592+0000 7f93d8723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:00.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.592+0000 7f93d8723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:00.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.592+0000 7f93d8723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:00.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.592+0000 7f93d8723640 1 -- 172.21.15.63:0/3051759945 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d00ade90 con 0x7f93d013fc60 2024-03-25T18:20:00.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.873+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93c8004e10 con 0x7f93d013fc60 2024-03-25T18:20:00.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.873+0000 7f93c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:00.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.873+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f93c8005120 con 0x7f93d013fc60 2024-03-25T18:20:00.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.873+0000 7f93c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:00.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.873+0000 7f93c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:00.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.873+0000 7f93c6ffd640 10 monclient: dump: 2024-03-25T18:20:00.874 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:00.874 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:00.874 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:00.874 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:00.874 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:00.875 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:00.875 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:00.875 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:00.875 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:00.875 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:00.875 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.873+0000 7f93d4cbc640 4 set_mon_vals no callback set 2024-03-25T18:20:00.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.874+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f93c801b0f0 con 0x7f93d013fc60 2024-03-25T18:20:00.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.875+0000 7f93d64bf640 10 monclient: get_auth_request con 0x7f93b406dd40 auth_method 0 2024-03-25T18:20:00.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.875+0000 7f93c6ffd640 10 client.4570.objecter ms_handle_connect 0x7f93b406dd40 2024-03-25T18:20:00.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.882+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f93c8054c20 con 0x7f93d013fc60 2024-03-25T18:20:00.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.883+0000 7f93c6ffd640 10 client.4570.objecter ms_dispatch 0x7f93d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:00.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.883+0000 7f93c6ffd640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:00.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.883+0000 7f93c6ffd640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:00.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.883+0000 7f93c6ffd640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T18:20:00.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.883+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f93c8050560 con 0x7f93d013fc60 2024-03-25T18:20:00.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.883+0000 7f93c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:00.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:00.883+0000 7f93c6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:01.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.139+0000 7f93d8723640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:01.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.139+0000 7f93d8723640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:01.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.139+0000 7f93d8723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:01.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.139+0000 7f93d8723640 1 -- 172.21.15.63:0/3051759945 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93d0141630 con 0x7f93d013fc60 2024-03-25T18:20:01.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.165+0000 7f93c6ffd640 1 -- 172.21.15.63:0/3051759945 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f93c8052660 con 0x7f93d013fc60 2024-03-25T18:20:01.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.165+0000 7f93c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:01.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.165+0000 7f93c6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:01.165 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:20:01.166 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:01.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.168+0000 7f93d8723640 10 librados: watch_flush enter 2024-03-25T18:20:01.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.168+0000 7f93d8723640 10 librados: watch_flush exit 2024-03-25T18:20:01.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.168+0000 7f93d8723640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T18:20:01.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.168+0000 7f93d8723640 10 client.4570.objecter successfully canceled tick 2024-03-25T18:20:01.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.168+0000 7f93d8723640 1 -- 172.21.15.63:0/3051759945 >> v1:172.21.15.81:6800/3640340873 conn(0x7f93b406dd40 legacy=0x7f93b4070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:01.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.168+0000 7f93d8723640 10 monclient: shutdown 2024-03-25T18:20:01.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.168+0000 7f93d8723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:01.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.168+0000 7f93d8723640 1 -- 172.21.15.63:0/3051759945 >> v1:172.21.15.81:6789/0 conn(0x7f93d013fc60 legacy=0x7f93d01d76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:01.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.169+0000 7f93d8723640 1 -- 172.21.15.63:0/3051759945 shutdown_connections 2024-03-25T18:20:01.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.169+0000 7f93d8723640 1 -- 172.21.15.63:0/3051759945 >> 172.21.15.63:0/3051759945 conn(0x7f93d00b4a80 msgr2=0x7f93d006b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:01.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.169+0000 7f93d8723640 1 -- 172.21.15.63:0/3051759945 shutdown_connections 2024-03-25T18:20:01.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.169+0000 7f93d8723640 1 -- 172.21.15.63:0/3051759945 wait complete. 2024-03-25T18:20:01.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:01.170+0000 7f93d8723640 1 librados: shutdown 2024-03-25T18:20:01.182 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:07.989 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:08.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.106+0000 7f762593e640 10 monclient: get_monmap_and_config 2024-03-25T18:20:08.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.106+0000 7f762593e640 10 monclient: build_initial_monmap 2024-03-25T18:20:08.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.106+0000 7f762593e640 10 monclient: monmap: 2024-03-25T18:20:08.107 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:20:08.108 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:08.108 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:20:08.107506+0000 2024-03-25T18:20:08.108 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:20:08.107506+0000 2024-03-25T18:20:08.108 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:08.108 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:08.108 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:20:08.108 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:20:08.108 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:20:08.108 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:08.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.107+0000 7f762593e640 1 Processor -- start 2024-03-25T18:20:08.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.107+0000 7f762593e640 1 -- start start 2024-03-25T18:20:08.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.107+0000 7f762593e640 10 monclient: init 2024-03-25T18:20:08.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.107+0000 7f762593e640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:08.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.107+0000 7f762593e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:08.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.108+0000 7f762593e640 10 monclient(hunting): picked mon.noname-a con 0x7f76200a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:20:08.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.108+0000 7f762593e640 10 monclient(hunting): picked mon.noname-b con 0x7f762013f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:20:08.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.108+0000 7f762593e640 10 monclient(hunting): picked mon.noname-c con 0x7f76201430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:20:08.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.108+0000 7f762593e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7620142520 con 0x7f76200a98f0 2024-03-25T18:20:08.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.108+0000 7f762593e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76200b28a0 con 0x7f76201430c0 2024-03-25T18:20:08.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.108+0000 7f762593e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76200b2a30 con 0x7f762013f710 2024-03-25T18:20:08.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.108+0000 7f762593e640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:08.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.108+0000 7f762593e640 10 monclient(hunting): authenticate will time out at 7902.294535s 2024-03-25T18:20:08.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.124+0000 7f761effd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f76200a98f0 0x7f76200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:40664/0 (socket says 172.21.15.63:40664) 2024-03-25T18:20:08.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.124+0000 7f761effd640 1 -- 172.21.15.63:0/78893098 learned_addr learned my addr 172.21.15.63:0/78893098 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:20:08.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.137+0000 7f761dffb640 1 -- 172.21.15.63:0/78893098 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242638781 0 0) 0x7f76200b28a0 con 0x7f76201430c0 2024-03-25T18:20:08.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.137+0000 7f761dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:08.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.137+0000 7f761dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:08.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.137+0000 7f761dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:08.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.137+0000 7f761dffb640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:20:08.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.137+0000 7f761dffb640 1 -- 172.21.15.63:0/78893098 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7608003520 con 0x7f76201430c0 2024-03-25T18:20:08.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.139+0000 7f761dffb640 1 -- 172.21.15.63:0/78893098 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460929758 0 0) 0x7f7620142520 con 0x7f76200a98f0 2024-03-25T18:20:08.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.139+0000 7f761dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:08.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.139+0000 7f761dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:08.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.139+0000 7f761dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:08.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.139+0000 7f761dffb640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:20:08.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.139+0000 7f761dffb640 1 -- 172.21.15.63:0/78893098 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76200b28a0 con 0x7f76200a98f0 2024-03-25T18:20:08.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f761dffb640 1 -- 172.21.15.63:0/78893098 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1612561826 0 0) 0x7f7608003520 con 0x7f76201430c0 2024-03-25T18:20:08.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f761dffb640 1 -- 172.21.15.63:0/78893098 >> v1:172.21.15.63:6790/0 conn(0x7f762013f710 legacy=0x7f76201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:20:08.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f761dffb640 1 -- 172.21.15.63:0/78893098 >> v1:172.21.15.63:6789/0 conn(0x7f76200a98f0 legacy=0x7f76200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:08.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f761dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:20:08.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f761dffb640 1 monclient: found mon.noname-c 2024-03-25T18:20:08.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f761dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:08.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f761dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:20:08.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f761dffb640 1 -- 172.21.15.63:0/78893098 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76200b44c0 con 0x7f76201430c0 2024-03-25T18:20:08.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f761dffb640 10 monclient: _finish_auth 0 2024-03-25T18:20:08.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f761dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:20:08.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f761dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:08.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f761dffb640 1 -- 172.21.15.63:0/78893098 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7618001a10 con 0x7f76201430c0 2024-03-25T18:20:08.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f762593e640 5 monclient: authenticate success, global_id 4573 2024-03-25T18:20:08.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f762593e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:08.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f761dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:08.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f761dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:08.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f761dffb640 10 monclient: dump: 2024-03-25T18:20:08.149 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:08.149 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:08.149 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:08.149 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:08.149 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:08.149 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:08.150 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:08.150 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:08.150 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:08.150 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:08.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.146+0000 7f762593e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:08.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.161+0000 7f761dffb640 1 -- 172.21.15.63:0/78893098 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7618001b70 con 0x7f76201430c0 2024-03-25T18:20:08.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.161+0000 7f761dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:08.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.161+0000 7f761dffb640 1 -- 172.21.15.63:0/78893098 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7618001f80 con 0x7f76201430c0 2024-03-25T18:20:08.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.161+0000 7f762593e640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:08.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.161+0000 7f762593e640 4 set_mon_vals no callback set 2024-03-25T18:20:08.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.161+0000 7f761dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:08.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.161+0000 7f761dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:08.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.161+0000 7f761dffb640 10 monclient: dump: 2024-03-25T18:20:08.163 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:08.163 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:08.163 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:08.163 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:08.163 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:08.163 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:08.163 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:08.163 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:08.163 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:08.163 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:08.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.162+0000 7f762593e640 10 monclient: shutdown 2024-03-25T18:20:08.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.162+0000 7f762593e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:08.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.162+0000 7f762593e640 1 -- 172.21.15.63:0/78893098 >> v1:172.21.15.81:6789/0 conn(0x7f76201430c0 legacy=0x7f762014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:08.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.162+0000 7f762593e640 1 -- 172.21.15.63:0/78893098 shutdown_connections 2024-03-25T18:20:08.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.162+0000 7f762593e640 1 -- 172.21.15.63:0/78893098 >> 172.21.15.63:0/78893098 conn(0x7f76200b7ea0 msgr2=0x7f76200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:08.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.162+0000 7f762593e640 1 -- 172.21.15.63:0/78893098 shutdown_connections 2024-03-25T18:20:08.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.162+0000 7f762593e640 1 -- 172.21.15.63:0/78893098 wait complete. 2024-03-25T18:20:08.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.163+0000 7f762593e640 10 monclient: build_initial_monmap 2024-03-25T18:20:08.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.163+0000 7f762593e640 10 monclient: monmap: 2024-03-25T18:20:08.166 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:20:08.166 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:08.166 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:20:08.166 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:20:08.166 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:08.166 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:20:08.166 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:20:08.166 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:20:08.166 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:20:08.166 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:08.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.163+0000 7f762593e640 1 librados: starting msgr at 2024-03-25T18:20:08.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.163+0000 7f762593e640 1 librados: starting objecter 2024-03-25T18:20:08.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.163+0000 7f762593e640 1 Processor -- start 2024-03-25T18:20:08.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.163+0000 7f762593e640 1 -- start start 2024-03-25T18:20:08.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.163+0000 7f762593e640 1 librados: setting wanted keys 2024-03-25T18:20:08.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.163+0000 7f762593e640 1 librados: calling monclient init 2024-03-25T18:20:08.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.163+0000 7f762593e640 10 monclient: init 2024-03-25T18:20:08.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.164+0000 7f762593e640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:08.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.164+0000 7f762593e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:08.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.164+0000 7f762593e640 10 monclient(hunting): picked mon.noname-c con 0x7f76200a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:20:08.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.164+0000 7f762593e640 10 monclient(hunting): picked mon.noname-b con 0x7f762013f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:20:08.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.164+0000 7f762593e640 10 monclient(hunting): picked mon.noname-a con 0x7f76201430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:20:08.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.164+0000 7f762593e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76200b1850 con 0x7f76201430c0 2024-03-25T18:20:08.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.164+0000 7f762593e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76200abaa0 con 0x7f762013f710 2024-03-25T18:20:08.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.164+0000 7f762593e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76200abd10 con 0x7f76200a98f0 2024-03-25T18:20:08.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.164+0000 7f762593e640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:08.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.164+0000 7f762593e640 10 monclient(hunting): authenticate will time out at 7902.350824s 2024-03-25T18:20:08.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.169+0000 7f761e7fc640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f762013f710 0x7f76200ac410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:60468/0 (socket says 172.21.15.63:60468) 2024-03-25T18:20:08.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.169+0000 7f761e7fc640 1 -- 172.21.15.63:0/1732617862 learned_addr learned my addr 172.21.15.63:0/1732617862 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:20:08.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.170+0000 7f7607fff640 10 client.?.objecter ms_handle_connect 0x7f76201430c0 2024-03-25T18:20:08.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.171+0000 7f7607fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:08.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.171+0000 7f7607fff640 10 client.?.objecter ms_handle_connect 0x7f762013f710 2024-03-25T18:20:08.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.171+0000 7f7607fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:08.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.179+0000 7f7607fff640 10 client.?.objecter ms_handle_connect 0x7f76200a98f0 2024-03-25T18:20:08.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.179+0000 7f7607fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:08.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.185+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739578376 0 0) 0x7f76200abaa0 con 0x7f762013f710 2024-03-25T18:20:08.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.185+0000 7f7607fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:08.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.185+0000 7f7607fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:08.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.185+0000 7f7607fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:08.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.185+0000 7f7607fff640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:20:08.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.185+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f760c004eb0 con 0x7f762013f710 2024-03-25T18:20:08.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.194+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870836692 0 0) 0x7f76200b1850 con 0x7f76201430c0 2024-03-25T18:20:08.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.194+0000 7f7607fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:08.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.194+0000 7f7607fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:08.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.194+0000 7f7607fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:08.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.194+0000 7f7607fff640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:20:08.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.194+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76200abaa0 con 0x7f76201430c0 2024-03-25T18:20:08.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.197+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081641800 0 0) 0x7f76200abd10 con 0x7f76200a98f0 2024-03-25T18:20:08.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.197+0000 7f7607fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:08.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.197+0000 7f7607fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:08.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.197+0000 7f7607fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:08.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.197+0000 7f7607fff640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:20:08.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.197+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76200b1850 con 0x7f76200a98f0 2024-03-25T18:20:08.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.197+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2040269289 0 0) 0x7f760c004eb0 con 0x7f762013f710 2024-03-25T18:20:08.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.198+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76200abd10 con 0x7f762013f710 2024-03-25T18:20:08.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.198+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7614004080 con 0x7f762013f710 2024-03-25T18:20:08.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.198+0000 7f7607fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:08.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.198+0000 7f7607fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:08.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.198+0000 7f7607fff640 10 monclient(hunting): dump: 2024-03-25T18:20:08.199 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:08.200 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:08.200 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:08.200 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:08.200 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:08.200 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:08.200 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:08.200 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:08.200 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:08.200 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:08.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.198+0000 7f7607fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:08.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 117257359 0 0) 0x7f76200abd10 con 0x7f762013f710 2024-03-25T18:20:08.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 >> v1:172.21.15.63:6790/0 conn(0x7f76200a98f0 legacy=0x7f76201387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:08.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 >> v1:172.21.15.63:6789/0 conn(0x7f76201430c0 legacy=0x7f76201e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:08.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f7607fff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:08.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f7607fff640 1 monclient: found mon.b 2024-03-25T18:20:08.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f7607fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:08.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f7607fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:08.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76200abff0 con 0x7f762013f710 2024-03-25T18:20:08.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f7607fff640 10 monclient: _finish_auth 0 2024-03-25T18:20:08.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f7607fff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:08.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f7607fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:08.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f762593e640 5 monclient: authenticate success, global_id 4576 2024-03-25T18:20:08.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f762593e640 10 monclient: _renew_subs 2024-03-25T18:20:08.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f762593e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:08.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f762593e640 1 -- 172.21.15.63:0/1732617862 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76200acb20 con 0x7f762013f710 2024-03-25T18:20:08.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f762593e640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:08.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f762593e640 10 monclient: _renew_subs 2024-03-25T18:20:08.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f762593e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:08.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f762593e640 1 -- 172.21.15.63:0/1732617862 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76201364e0 con 0x7f762013f710 2024-03-25T18:20:08.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.206+0000 7f762593e640 1 librados: init done 2024-03-25T18:20:08.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.207+0000 7f762593e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:08.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.207+0000 7f762593e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:08.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.207+0000 7f762593e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:08.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.207+0000 7f762593e640 1 -- 172.21.15.63:0/1732617862 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76200627b0 con 0x7f762013f710 2024-03-25T18:20:08.475 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.474+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7614000ed0 con 0x7f762013f710 2024-03-25T18:20:08.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.474+0000 7f7607fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:08.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.474+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f76140012a0 con 0x7f762013f710 2024-03-25T18:20:08.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.474+0000 7f7607fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:08.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.474+0000 7f7607fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:08.476 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.474+0000 7f7607fff640 10 monclient: dump: 2024-03-25T18:20:08.476 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:08.476 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:08.476 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:08.476 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:08.476 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:08.476 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:08.476 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:08.476 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:08.476 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:08.477 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:08.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.474+0000 7f761d7fa640 4 set_mon_vals no callback set 2024-03-25T18:20:08.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.475+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f76140014c0 con 0x7f762013f710 2024-03-25T18:20:08.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.476+0000 7f761effd640 10 monclient: get_auth_request con 0x7f760c06ddf0 auth_method 0 2024-03-25T18:20:08.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.476+0000 7f7607fff640 10 client.4576.objecter ms_handle_connect 0x7f760c06ddf0 2024-03-25T18:20:08.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.480+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f7614054800 con 0x7f762013f710 2024-03-25T18:20:08.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.480+0000 7f7607fff640 10 client.4576.objecter ms_dispatch 0x7f7620000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:08.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.480+0000 7f7607fff640 3 client.4576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:08.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.480+0000 7f7607fff640 3 client.4576.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:08.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.481+0000 7f7607fff640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-25T18:20:08.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.481+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7614050310 con 0x7f762013f710 2024-03-25T18:20:08.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.481+0000 7f7607fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:08.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.481+0000 7f7607fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:08.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.705+0000 7f762593e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:08.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.705+0000 7f762593e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:08.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.705+0000 7f762593e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:08.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.705+0000 7f762593e640 1 -- 172.21.15.63:0/1732617862 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f762013dbb0 con 0x7f762013f710 2024-03-25T18:20:08.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.724+0000 7f7607fff640 1 -- 172.21.15.63:0/1732617862 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f7614052240 con 0x7f762013f710 2024-03-25T18:20:08.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.724+0000 7f7607fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:08.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.724+0000 7f7607fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:08.726 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:20:08.726 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:08.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.726+0000 7f762593e640 10 librados: watch_flush enter 2024-03-25T18:20:08.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.726+0000 7f762593e640 10 librados: watch_flush exit 2024-03-25T18:20:08.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.726+0000 7f762593e640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-25T18:20:08.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.726+0000 7f762593e640 10 client.4576.objecter successfully canceled tick 2024-03-25T18:20:08.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.727+0000 7f762593e640 1 -- 172.21.15.63:0/1732617862 >> v1:172.21.15.81:6800/3640340873 conn(0x7f760c06ddf0 legacy=0x7f760c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:08.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.727+0000 7f762593e640 10 monclient: shutdown 2024-03-25T18:20:08.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.727+0000 7f762593e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:08.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.727+0000 7f762593e640 1 -- 172.21.15.63:0/1732617862 >> v1:172.21.15.81:6789/0 conn(0x7f762013f710 legacy=0x7f76200ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:08.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.727+0000 7f762593e640 1 -- 172.21.15.63:0/1732617862 shutdown_connections 2024-03-25T18:20:08.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.727+0000 7f762593e640 1 -- 172.21.15.63:0/1732617862 >> 172.21.15.63:0/1732617862 conn(0x7f76200b7ea0 msgr2=0x7f7620074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:08.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.727+0000 7f762593e640 1 -- 172.21.15.63:0/1732617862 shutdown_connections 2024-03-25T18:20:08.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.727+0000 7f762593e640 1 -- 172.21.15.63:0/1732617862 wait complete. 2024-03-25T18:20:08.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:08.727+0000 7f762593e640 1 librados: shutdown 2024-03-25T18:20:08.740 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:15.546 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:15.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.661+0000 7f5c05063640 10 monclient: get_monmap_and_config 2024-03-25T18:20:15.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.661+0000 7f5c05063640 10 monclient: build_initial_monmap 2024-03-25T18:20:15.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.661+0000 7f5c05063640 10 monclient: monmap: 2024-03-25T18:20:15.663 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:20:15.663 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:15.663 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:20:15.663059+0000 2024-03-25T18:20:15.663 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:20:15.663059+0000 2024-03-25T18:20:15.663 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:15.663 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:15.664 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:20:15.664 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:20:15.664 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:20:15.664 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:15.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.662+0000 7f5c05063640 1 Processor -- start 2024-03-25T18:20:15.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.662+0000 7f5c05063640 1 -- start start 2024-03-25T18:20:15.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.662+0000 7f5c05063640 10 monclient: init 2024-03-25T18:20:15.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.663+0000 7f5c05063640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:15.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.663+0000 7f5c05063640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:15.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.663+0000 7f5c05063640 10 monclient(hunting): picked mon.noname-b con 0x7f5c000a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:20:15.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.663+0000 7f5c05063640 10 monclient(hunting): picked mon.noname-c con 0x7f5c0013f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:20:15.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.663+0000 7f5c05063640 10 monclient(hunting): picked mon.noname-a con 0x7f5c001430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:20:15.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.663+0000 7f5c05063640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c00142520 con 0x7f5c001430c0 2024-03-25T18:20:15.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.663+0000 7f5c05063640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c000b28a0 con 0x7f5c0013f710 2024-03-25T18:20:15.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.663+0000 7f5c05063640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c000b2a30 con 0x7f5c000a98f0 2024-03-25T18:20:15.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.663+0000 7f5c05063640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:15.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.663+0000 7f5c05063640 10 monclient(hunting): authenticate will time out at 7909.849929s 2024-03-25T18:20:15.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.670+0000 7f5bfed9d640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f5c000a98f0 0x7f5c000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:59846/0 (socket says 172.21.15.63:59846) 2024-03-25T18:20:15.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.670+0000 7f5bfed9d640 1 -- 172.21.15.63:0/588653701 learned_addr learned my addr 172.21.15.63:0/588653701 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:20:15.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.687+0000 7f5bfdd9b640 1 -- 172.21.15.63:0/588653701 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570480582 0 0) 0x7f5c000b2a30 con 0x7f5c000a98f0 2024-03-25T18:20:15.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.687+0000 7f5bfdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:15.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.687+0000 7f5bfdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:15.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.687+0000 7f5bfdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:15.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.687+0000 7f5bfdd9b640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:20:15.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.687+0000 7f5bfdd9b640 1 -- 172.21.15.63:0/588653701 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5be8003520 con 0x7f5c000a98f0 2024-03-25T18:20:15.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.687+0000 7f5bfdd9b640 1 -- 172.21.15.63:0/588653701 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678159405 0 0) 0x7f5c000b28a0 con 0x7f5c0013f710 2024-03-25T18:20:15.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.687+0000 7f5bfdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:15.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.687+0000 7f5bfdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:15.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.687+0000 7f5bfdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:15.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.687+0000 7f5bfdd9b640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:20:15.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.687+0000 7f5bfdd9b640 1 -- 172.21.15.63:0/588653701 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c000b2a30 con 0x7f5c0013f710 2024-03-25T18:20:15.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5bfdd9b640 1 -- 172.21.15.63:0/588653701 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 968763656 0 0) 0x7f5be8003520 con 0x7f5c000a98f0 2024-03-25T18:20:15.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5bfdd9b640 1 -- 172.21.15.63:0/588653701 >> v1:172.21.15.81:6789/0 conn(0x7f5c0013f710 legacy=0x7f5c001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:15.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5bfdd9b640 1 -- 172.21.15.63:0/588653701 >> v1:172.21.15.63:6789/0 conn(0x7f5c001430c0 legacy=0x7f5c0014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:15.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5bfdd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:20:15.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5bfdd9b640 1 monclient: found mon.noname-b 2024-03-25T18:20:15.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5bfdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:15.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5bfdd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:20:15.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5bfdd9b640 1 -- 172.21.15.63:0/588653701 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c000b44c0 con 0x7f5c000a98f0 2024-03-25T18:20:15.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5bfdd9b640 10 monclient: _finish_auth 0 2024-03-25T18:20:15.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5bfdd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:20:15.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5bfdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:15.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5c05063640 5 monclient: authenticate success, global_id 4553 2024-03-25T18:20:15.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5c05063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:15.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5bfdd9b640 1 -- 172.21.15.63:0/588653701 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5bf0001650 con 0x7f5c000a98f0 2024-03-25T18:20:15.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5bfdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:15.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5bfdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:15.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5bfdd9b640 10 monclient: dump: 2024-03-25T18:20:15.700 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:15.700 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:15.700 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:15.700 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:15.700 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:15.700 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:15.700 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:15.700 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:15.700 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:15.700 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:15.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.697+0000 7f5c05063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:15.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.708+0000 7f5bfdd9b640 1 -- 172.21.15.63:0/588653701 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bf00018a0 con 0x7f5c000a98f0 2024-03-25T18:20:15.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.708+0000 7f5bfdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:15.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.708+0000 7f5bfdd9b640 1 -- 172.21.15.63:0/588653701 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5bf0001c30 con 0x7f5c000a98f0 2024-03-25T18:20:15.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.708+0000 7f5c05063640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:15.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.708+0000 7f5c05063640 4 set_mon_vals no callback set 2024-03-25T18:20:15.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.708+0000 7f5bfdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:15.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.708+0000 7f5bfdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:15.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.708+0000 7f5bfdd9b640 10 monclient: dump: 2024-03-25T18:20:15.710 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:15.710 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:15.710 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:15.710 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:15.710 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:15.710 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:15.710 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:15.710 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:15.711 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:15.711 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:15.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.709+0000 7f5c05063640 10 monclient: shutdown 2024-03-25T18:20:15.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.709+0000 7f5c05063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:15.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.709+0000 7f5c05063640 1 -- 172.21.15.63:0/588653701 >> v1:172.21.15.63:6790/0 conn(0x7f5c000a98f0 legacy=0x7f5c000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:15.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.709+0000 7f5c05063640 1 -- 172.21.15.63:0/588653701 shutdown_connections 2024-03-25T18:20:15.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.709+0000 7f5c05063640 1 -- 172.21.15.63:0/588653701 >> 172.21.15.63:0/588653701 conn(0x7f5c000b7ea0 msgr2=0x7f5c000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:15.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.709+0000 7f5c05063640 1 -- 172.21.15.63:0/588653701 shutdown_connections 2024-03-25T18:20:15.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.709+0000 7f5c05063640 1 -- 172.21.15.63:0/588653701 wait complete. 2024-03-25T18:20:15.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.710+0000 7f5c05063640 10 monclient: build_initial_monmap 2024-03-25T18:20:15.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.710+0000 7f5c05063640 10 monclient: monmap: 2024-03-25T18:20:15.713 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:20:15.713 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:15.713 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:20:15.713 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:20:15.713 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:15.713 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:20:15.713 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:20:15.713 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:20:15.713 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:20:15.713 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:15.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.710+0000 7f5c05063640 1 librados: starting msgr at 2024-03-25T18:20:15.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.710+0000 7f5c05063640 1 librados: starting objecter 2024-03-25T18:20:15.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.711+0000 7f5c05063640 1 Processor -- start 2024-03-25T18:20:15.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.711+0000 7f5c05063640 1 -- start start 2024-03-25T18:20:15.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.711+0000 7f5c05063640 1 librados: setting wanted keys 2024-03-25T18:20:15.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.711+0000 7f5c05063640 1 librados: calling monclient init 2024-03-25T18:20:15.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.711+0000 7f5c05063640 10 monclient: init 2024-03-25T18:20:15.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.711+0000 7f5c05063640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:15.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.711+0000 7f5c05063640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:15.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.711+0000 7f5c05063640 10 monclient(hunting): picked mon.noname-a con 0x7f5c000a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:20:15.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.711+0000 7f5c05063640 10 monclient(hunting): picked mon.noname-c con 0x7f5c0013f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:20:15.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.712+0000 7f5c05063640 10 monclient(hunting): picked mon.noname-b con 0x7f5c001430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:20:15.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.712+0000 7f5c05063640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c000abd10 con 0x7f5c000a98f0 2024-03-25T18:20:15.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.712+0000 7f5c05063640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c001368b0 con 0x7f5c001430c0 2024-03-25T18:20:15.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.712+0000 7f5c05063640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c00137ee0 con 0x7f5c0013f710 2024-03-25T18:20:15.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.712+0000 7f5c05063640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:15.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.712+0000 7f5c05063640 10 monclient(hunting): authenticate will time out at 7909.898371s 2024-03-25T18:20:15.721 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.720+0000 7f5bfed9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f5c000a98f0 0x7f5c000ac410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:43004/0 (socket says 172.21.15.63:43004) 2024-03-25T18:20:15.721 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.720+0000 7f5bfed9d640 1 -- 172.21.15.63:0/1823966383 learned_addr learned my addr 172.21.15.63:0/1823966383 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:20:15.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.724+0000 7f5be77fe640 10 client.?.objecter ms_handle_connect 0x7f5c000a98f0 2024-03-25T18:20:15.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.724+0000 7f5be77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:15.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.733+0000 7f5be77fe640 10 client.?.objecter ms_handle_connect 0x7f5c0013f710 2024-03-25T18:20:15.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.733+0000 7f5be77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:15.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.734+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442359890 0 0) 0x7f5c000abd10 con 0x7f5c000a98f0 2024-03-25T18:20:15.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.734+0000 7f5be77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:15.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.734+0000 7f5be77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:15.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.734+0000 7f5be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:15.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.734+0000 7f5be77fe640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:20:15.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.734+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bec004eb0 con 0x7f5c000a98f0 2024-03-25T18:20:15.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.738+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970606952 0 0) 0x7f5c00137ee0 con 0x7f5c0013f710 2024-03-25T18:20:15.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.738+0000 7f5be77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:15.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.738+0000 7f5be77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:15.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.738+0000 7f5be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:15.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.738+0000 7f5be77fe640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:20:15.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.738+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c000abd10 con 0x7f5c0013f710 2024-03-25T18:20:15.748 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.747+0000 7f5be77fe640 10 client.?.objecter ms_handle_connect 0x7f5c001430c0 2024-03-25T18:20:15.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.747+0000 7f5be77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:15.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.748+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 314679823 0 0) 0x7f5c000abd10 con 0x7f5c0013f710 2024-03-25T18:20:15.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.748+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c00137ee0 con 0x7f5c0013f710 2024-03-25T18:20:15.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.752+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5bf40019b0 con 0x7f5c0013f710 2024-03-25T18:20:15.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.752+0000 7f5be77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:15.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.752+0000 7f5be77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:15.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.752+0000 7f5be77fe640 10 monclient(hunting): dump: 2024-03-25T18:20:15.754 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:15.754 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:15.754 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:15.754 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:15.754 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:15.754 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:15.754 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:15.754 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:15.754 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:15.754 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:15.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.752+0000 7f5be77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:15.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.755+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765693663 0 0) 0x7f5c001368b0 con 0x7f5c001430c0 2024-03-25T18:20:15.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.755+0000 7f5be77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:15.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.755+0000 7f5be77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:15.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.755+0000 7f5be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:15.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.755+0000 7f5be77fe640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:20:15.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.755+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c000abd10 con 0x7f5c001430c0 2024-03-25T18:20:15.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3961769662 0 0) 0x7f5bec004eb0 con 0x7f5c000a98f0 2024-03-25T18:20:15.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c001368b0 con 0x7f5c000a98f0 2024-03-25T18:20:15.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5bf0001bb0 con 0x7f5c000a98f0 2024-03-25T18:20:15.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:15.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:15.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 10 monclient(hunting): dump: 2024-03-25T18:20:15.764 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:15.764 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:15.764 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:15.765 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:15.765 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:15.765 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:15.765 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:15.765 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:15.765 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:15.765 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:15.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1151965315 0 0) 0x7f5c00137ee0 con 0x7f5c0013f710 2024-03-25T18:20:15.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 >> v1:172.21.15.81:6789/0 conn(0x7f5c001430c0 legacy=0x7f5c00135a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:15.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 >> v1:172.21.15.63:6789/0 conn(0x7f5c000a98f0 legacy=0x7f5c000ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:15.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:15.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 1 monclient: found mon.c 2024-03-25T18:20:15.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:15.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:20:15.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c00138070 con 0x7f5c0013f710 2024-03-25T18:20:15.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:15.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:15.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5be77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:15.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5c05063640 5 monclient: authenticate success, global_id 4556 2024-03-25T18:20:15.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5c05063640 10 monclient: _renew_subs 2024-03-25T18:20:15.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5c05063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:20:15.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5c05063640 1 -- 172.21.15.63:0/1823966383 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c00139380 con 0x7f5c0013f710 2024-03-25T18:20:15.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5c05063640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:15.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5c05063640 10 monclient: _renew_subs 2024-03-25T18:20:15.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5c05063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:20:15.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5c05063640 1 -- 172.21.15.63:0/1823966383 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c00138680 con 0x7f5c0013f710 2024-03-25T18:20:15.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.762+0000 7f5c05063640 1 librados: init done 2024-03-25T18:20:15.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.763+0000 7f5c05063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:15.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.763+0000 7f5c05063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:15.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.763+0000 7f5c05063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:20:15.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:15.763+0000 7f5c05063640 1 -- 172.21.15.63:0/1823966383 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c000627b0 con 0x7f5c0013f710 2024-03-25T18:20:16.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.079+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bf4003610 con 0x7f5c0013f710 2024-03-25T18:20:16.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.079+0000 7f5be77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:16.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.079+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f5bf4003a20 con 0x7f5c0013f710 2024-03-25T18:20:16.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.079+0000 7f5be77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:16.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.079+0000 7f5be77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:16.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.079+0000 7f5be77fe640 10 monclient: dump: 2024-03-25T18:20:16.081 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:16.081 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:16.081 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:16.082 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:16.082 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:16.082 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:16.082 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:16.082 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:16.082 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:16.082 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:16.082 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.079+0000 7f5bfd59a640 4 set_mon_vals no callback set 2024-03-25T18:20:16.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.080+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f5bf4003c40 con 0x7f5c0013f710 2024-03-25T18:20:16.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.081+0000 7f5bfed9d640 10 monclient: get_auth_request con 0x7f5bec06df10 auth_method 0 2024-03-25T18:20:16.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.081+0000 7f5be77fe640 10 client.4556.objecter ms_handle_connect 0x7f5bec06df10 2024-03-25T18:20:16.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.087+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f5bf4053610 con 0x7f5c0013f710 2024-03-25T18:20:16.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.087+0000 7f5be77fe640 10 client.4556.objecter ms_dispatch 0x7f5c00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:16.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.087+0000 7f5be77fe640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:16.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.087+0000 7f5be77fe640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:16.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.087+0000 7f5be77fe640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-25T18:20:16.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.087+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5bf404f120 con 0x7f5c0013f710 2024-03-25T18:20:16.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.087+0000 7f5be77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:16.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.087+0000 7f5be77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:16.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.352+0000 7f5c05063640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:16.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.352+0000 7f5c05063640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:16.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.352+0000 7f5c05063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:20:16.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.352+0000 7f5c05063640 1 -- 172.21.15.63:0/1823966383 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c00139510 con 0x7f5c0013f710 2024-03-25T18:20:16.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.394+0000 7f5be77fe640 1 -- 172.21.15.63:0/1823966383 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f5bf4051050 con 0x7f5c0013f710 2024-03-25T18:20:16.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.394+0000 7f5be77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:16.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.394+0000 7f5be77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:16.396 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:20:16.396 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:16.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.396+0000 7f5c05063640 10 librados: watch_flush enter 2024-03-25T18:20:16.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.396+0000 7f5c05063640 10 librados: watch_flush exit 2024-03-25T18:20:16.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.396+0000 7f5c05063640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-25T18:20:16.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.396+0000 7f5c05063640 10 client.4556.objecter successfully canceled tick 2024-03-25T18:20:16.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.396+0000 7f5c05063640 1 -- 172.21.15.63:0/1823966383 >> v1:172.21.15.81:6800/3640340873 conn(0x7f5bec06df10 legacy=0x7f5bec0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:16.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.397+0000 7f5c05063640 10 monclient: shutdown 2024-03-25T18:20:16.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.397+0000 7f5c05063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:16.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.397+0000 7f5c05063640 1 -- 172.21.15.63:0/1823966383 >> v1:172.21.15.63:6790/0 conn(0x7f5c0013f710 legacy=0x7f5c001352f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:16.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.397+0000 7f5c05063640 1 -- 172.21.15.63:0/1823966383 shutdown_connections 2024-03-25T18:20:16.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.397+0000 7f5c05063640 1 -- 172.21.15.63:0/1823966383 >> 172.21.15.63:0/1823966383 conn(0x7f5c000b7ea0 msgr2=0x7f5c00074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:16.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.397+0000 7f5c05063640 1 -- 172.21.15.63:0/1823966383 shutdown_connections 2024-03-25T18:20:16.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.397+0000 7f5c05063640 1 -- 172.21.15.63:0/1823966383 wait complete. 2024-03-25T18:20:16.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:16.397+0000 7f5c05063640 1 librados: shutdown 2024-03-25T18:20:16.411 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:19.509 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:01.201 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:19.515 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:01.207 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:19.549 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:01.241 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:19.556 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:01.248 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:19.563 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:08:01.223 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:19.572 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:08:01.232 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:19.573 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:01.265 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:19.608 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:01.300 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:19.618 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:08:01.279 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:23.218 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:23.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.330+0000 7f980f063640 10 monclient: get_monmap_and_config 2024-03-25T18:20:23.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.330+0000 7f980f063640 10 monclient: build_initial_monmap 2024-03-25T18:20:23.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.330+0000 7f980f063640 10 monclient: monmap: 2024-03-25T18:20:23.332 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:20:23.332 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:23.332 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:20:23.331702+0000 2024-03-25T18:20:23.332 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:20:23.331702+0000 2024-03-25T18:20:23.332 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:23.332 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:23.332 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:20:23.332 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:20:23.332 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:20:23.332 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:23.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.331+0000 7f980f063640 1 Processor -- start 2024-03-25T18:20:23.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.331+0000 7f980f063640 1 -- start start 2024-03-25T18:20:23.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.331+0000 7f980f063640 10 monclient: init 2024-03-25T18:20:23.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.332+0000 7f980f063640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:23.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.332+0000 7f980f063640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:23.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.332+0000 7f980f063640 10 monclient(hunting): picked mon.noname-b con 0x7f98080704a0 addr v1:172.21.15.63:6790/0 2024-03-25T18:20:23.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.332+0000 7f980f063640 10 monclient(hunting): picked mon.noname-a con 0x7f9808070880 addr v1:172.21.15.63:6789/0 2024-03-25T18:20:23.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.332+0000 7f980f063640 10 monclient(hunting): picked mon.noname-c con 0x7f9808149a40 addr v1:172.21.15.81:6789/0 2024-03-25T18:20:23.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.332+0000 7f980f063640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9808143870 con 0x7f9808070880 2024-03-25T18:20:23.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.332+0000 7f980f063640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9808072480 con 0x7f9808149a40 2024-03-25T18:20:23.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.332+0000 7f980f063640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9808072760 con 0x7f98080704a0 2024-03-25T18:20:23.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.332+0000 7f980f063640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:23.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.332+0000 7f980f063640 10 monclient(hunting): authenticate will time out at 7917.518873s 2024-03-25T18:20:23.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.333+0000 7f980cdff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f98080704a0 0x7f9808143d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:54884/0 (socket says 172.21.15.63:54884) 2024-03-25T18:20:23.335 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.333+0000 7f980cdff640 1 -- 172.21.15.63:0/2420950629 learned_addr learned my addr 172.21.15.63:0/2420950629 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:20:23.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.388+0000 7f98077fe640 1 -- 172.21.15.63:0/2420950629 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432666225 0 0) 0x7f9808143870 con 0x7f9808070880 2024-03-25T18:20:23.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.388+0000 7f98077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:23.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.388+0000 7f98077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:23.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.388+0000 7f98077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:23.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.388+0000 7f98077fe640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:20:23.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.388+0000 7f98077fe640 1 -- 172.21.15.63:0/2420950629 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97f8003520 con 0x7f9808070880 2024-03-25T18:20:23.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f98077fe640 1 -- 172.21.15.63:0/2420950629 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1844607826 0 0) 0x7f97f8003520 con 0x7f9808070880 2024-03-25T18:20:23.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f98077fe640 1 -- 172.21.15.63:0/2420950629 >> v1:172.21.15.63:6790/0 conn(0x7f98080704a0 legacy=0x7f9808143d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:23.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f98077fe640 1 -- 172.21.15.63:0/2420950629 >> v1:172.21.15.81:6789/0 conn(0x7f9808149a40 legacy=0x7f980814be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:23.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f98077fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:23.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f98077fe640 1 monclient: found mon.noname-a 2024-03-25T18:20:23.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f98077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:23.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f98077fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:20:23.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f98077fe640 1 -- 172.21.15.63:0/2420950629 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9808072a40 con 0x7f9808070880 2024-03-25T18:20:23.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f98077fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:23.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f98077fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:23.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f98077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:23.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f98077fe640 1 -- 172.21.15.63:0/2420950629 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f97f4001690 con 0x7f9808070880 2024-03-25T18:20:23.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f98077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:23.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f98077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:23.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f98077fe640 10 monclient: dump: 2024-03-25T18:20:23.404 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:23.404 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:23.404 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:23.404 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:23.404 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:23.404 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:23.404 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:23.404 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:23.404 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:23.404 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:23.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f980f063640 5 monclient: authenticate success, global_id 4557 2024-03-25T18:20:23.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.401+0000 7f980f063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:23.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.415+0000 7f98077fe640 1 -- 172.21.15.63:0/2420950629 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97f40017f0 con 0x7f9808070880 2024-03-25T18:20:23.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.415+0000 7f98077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:23.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.415+0000 7f98077fe640 1 -- 172.21.15.63:0/2420950629 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f97f4001bc0 con 0x7f9808070880 2024-03-25T18:20:23.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.415+0000 7f980f063640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:23.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.415+0000 7f980f063640 4 set_mon_vals no callback set 2024-03-25T18:20:23.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.415+0000 7f98077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:23.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.415+0000 7f98077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:23.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.415+0000 7f98077fe640 10 monclient: dump: 2024-03-25T18:20:23.417 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:23.417 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:23.417 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:23.417 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:23.417 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:23.417 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:23.418 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:23.418 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:23.418 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:23.418 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:23.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.416+0000 7f980f063640 10 monclient: shutdown 2024-03-25T18:20:23.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.416+0000 7f980f063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:23.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.416+0000 7f980f063640 1 -- 172.21.15.63:0/2420950629 >> v1:172.21.15.63:6789/0 conn(0x7f9808070880 legacy=0x7f980813d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:23.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.416+0000 7f980f063640 1 -- 172.21.15.63:0/2420950629 shutdown_connections 2024-03-25T18:20:23.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.416+0000 7f980f063640 1 -- 172.21.15.63:0/2420950629 >> 172.21.15.63:0/2420950629 conn(0x7f98080b5c00 msgr2=0x7f98080b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:23.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.416+0000 7f980f063640 1 -- 172.21.15.63:0/2420950629 shutdown_connections 2024-03-25T18:20:23.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.416+0000 7f980f063640 1 -- 172.21.15.63:0/2420950629 wait complete. 2024-03-25T18:20:23.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.417+0000 7f980f063640 10 monclient: build_initial_monmap 2024-03-25T18:20:23.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.417+0000 7f980f063640 10 monclient: monmap: 2024-03-25T18:20:23.420 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:20:23.420 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:23.420 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:20:23.420 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:20:23.421 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:23.421 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:20:23.421 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:20:23.421 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:20:23.421 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:20:23.421 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:23.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.417+0000 7f980f063640 1 librados: starting msgr at 2024-03-25T18:20:23.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.417+0000 7f980f063640 1 librados: starting objecter 2024-03-25T18:20:23.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.417+0000 7f980f063640 1 Processor -- start 2024-03-25T18:20:23.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.417+0000 7f980f063640 1 -- start start 2024-03-25T18:20:23.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.417+0000 7f980f063640 1 librados: setting wanted keys 2024-03-25T18:20:23.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.417+0000 7f980f063640 1 librados: calling monclient init 2024-03-25T18:20:23.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.417+0000 7f980f063640 10 monclient: init 2024-03-25T18:20:23.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.418+0000 7f980f063640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:23.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.418+0000 7f980f063640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:23.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.418+0000 7f980f063640 10 monclient(hunting): picked mon.noname-c con 0x7f98080704a0 addr v1:172.21.15.63:6790/0 2024-03-25T18:20:23.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.418+0000 7f980f063640 10 monclient(hunting): picked mon.noname-b con 0x7f9808070880 addr v1:172.21.15.81:6789/0 2024-03-25T18:20:23.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.418+0000 7f980f063640 10 monclient(hunting): picked mon.noname-a con 0x7f9808149a40 addr v1:172.21.15.63:6789/0 2024-03-25T18:20:23.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.418+0000 7f980f063640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98081d5240 con 0x7f9808149a40 2024-03-25T18:20:23.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.418+0000 7f980f063640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98081d5520 con 0x7f9808070880 2024-03-25T18:20:23.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.418+0000 7f980f063640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98081d60c0 con 0x7f98080704a0 2024-03-25T18:20:23.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.418+0000 7f980f063640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:23.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.418+0000 7f980f063640 10 monclient(hunting): authenticate will time out at 7917.605078s 2024-03-25T18:20:23.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.425+0000 7f980d600640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f9808149a40 0x7f9808139f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:54810/0 (socket says 172.21.15.63:54810) 2024-03-25T18:20:23.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.425+0000 7f980d600640 1 -- 172.21.15.63:0/3243089904 learned_addr learned my addr 172.21.15.63:0/3243089904 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:20:23.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.427+0000 7f98057fa640 10 client.?.objecter ms_handle_connect 0x7f9808149a40 2024-03-25T18:20:23.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.427+0000 7f98057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:23.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.433+0000 7f98057fa640 10 client.?.objecter ms_handle_connect 0x7f9808070880 2024-03-25T18:20:23.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.433+0000 7f98057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:23.445 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.444+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235427105 0 0) 0x7f98081d5240 con 0x7f9808149a40 2024-03-25T18:20:23.445 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.444+0000 7f98057fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:23.445 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.444+0000 7f98057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:23.445 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.444+0000 7f98057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:23.445 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.444+0000 7f98057fa640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:20:23.445 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.444+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97ec004eb0 con 0x7f9808149a40 2024-03-25T18:20:23.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.453+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143605836 0 0) 0x7f98081d5520 con 0x7f9808070880 2024-03-25T18:20:23.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.453+0000 7f98057fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:23.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.453+0000 7f98057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:23.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.453+0000 7f98057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:23.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.453+0000 7f98057fa640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:20:23.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.453+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98081d5240 con 0x7f9808070880 2024-03-25T18:20:23.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.461+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3798975910 0 0) 0x7f97ec004eb0 con 0x7f9808149a40 2024-03-25T18:20:23.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.461+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98081d5520 con 0x7f9808149a40 2024-03-25T18:20:23.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.461+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9800001c30 con 0x7f9808149a40 2024-03-25T18:20:23.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.461+0000 7f98057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:23.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.461+0000 7f98057fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:23.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.461+0000 7f98057fa640 10 monclient(hunting): dump: 2024-03-25T18:20:23.463 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:23.463 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:23.463 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:23.463 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:23.463 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:23.463 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:23.463 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:23.463 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:23.463 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:23.463 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:23.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.461+0000 7f98057fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:23.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.469+0000 7f98057fa640 10 client.?.objecter ms_handle_connect 0x7f98080704a0 2024-03-25T18:20:23.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.469+0000 7f98057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:23.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.471+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3417109730 0 0) 0x7f98081d5240 con 0x7f9808070880 2024-03-25T18:20:23.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.471+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97ec004eb0 con 0x7f9808070880 2024-03-25T18:20:23.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.471+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f97f4001b00 con 0x7f9808070880 2024-03-25T18:20:23.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.471+0000 7f98057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:23.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.471+0000 7f98057fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:23.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.471+0000 7f98057fa640 10 monclient(hunting): dump: 2024-03-25T18:20:23.472 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:23.472 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:23.472 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:23.473 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:23.473 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:23.473 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:23.473 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:23.473 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:23.473 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:23.473 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:23.480 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.478+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3879336685 0 0) 0x7f98081d5520 con 0x7f9808149a40 2024-03-25T18:20:23.480 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.478+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 >> v1:172.21.15.63:6790/0 conn(0x7f98080704a0 legacy=0x7f98081d59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:23.480 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 >> v1:172.21.15.81:6789/0 conn(0x7f9808070880 legacy=0x7f9808142c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:23.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f98057fa640 10 monclient: _finish_hunting 0 2024-03-25T18:20:23.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f98057fa640 1 monclient: found mon.a 2024-03-25T18:20:23.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f98057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:23.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f98057fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:20:23.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98081387e0 con 0x7f9808149a40 2024-03-25T18:20:23.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f98057fa640 10 monclient: _finish_auth 0 2024-03-25T18:20:23.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f98057fa640 10 monclient: _check_auth_tickets 2024-03-25T18:20:23.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f98057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:23.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f980f063640 5 monclient: authenticate success, global_id 4560 2024-03-25T18:20:23.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f980f063640 10 monclient: _renew_subs 2024-03-25T18:20:23.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f980f063640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:20:23.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f980f063640 1 -- 172.21.15.63:0/3243089904 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9808143390 con 0x7f9808149a40 2024-03-25T18:20:23.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f980f063640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:23.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f980f063640 10 monclient: _renew_subs 2024-03-25T18:20:23.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f980f063640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:20:23.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f980f063640 1 -- 172.21.15.63:0/3243089904 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f980812fa70 con 0x7f9808149a40 2024-03-25T18:20:23.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.479+0000 7f980f063640 1 librados: init done 2024-03-25T18:20:23.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.480+0000 7f980f063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:23.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.480+0000 7f980f063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:23.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.480+0000 7f980f063640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:20:23.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.480+0000 7f980f063640 1 -- 172.21.15.63:0/3243089904 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98080627b0 con 0x7f9808149a40 2024-03-25T18:20:23.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.796+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9800003990 con 0x7f9808149a40 2024-03-25T18:20:23.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.796+0000 7f98057fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:23.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.796+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9800003d20 con 0x7f9808149a40 2024-03-25T18:20:23.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.796+0000 7f98057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:23.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.796+0000 7f98057fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:23.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.796+0000 7f98057fa640 10 monclient: dump: 2024-03-25T18:20:23.798 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:23.798 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:23.798 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:23.798 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:23.798 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:23.798 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:23.799 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:23.799 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:23.799 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:23.799 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:23.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.796+0000 7f9806ffd640 4 set_mon_vals no callback set 2024-03-25T18:20:23.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.797+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f9800003f40 con 0x7f9808149a40 2024-03-25T18:20:23.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.797+0000 7f980cdff640 10 monclient: get_auth_request con 0x7f97ec06df60 auth_method 0 2024-03-25T18:20:23.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.798+0000 7f98057fa640 10 client.4560.objecter ms_handle_connect 0x7f97ec06df60 2024-03-25T18:20:23.805 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.803+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f98000549e0 con 0x7f9808149a40 2024-03-25T18:20:23.805 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.803+0000 7f98057fa640 10 client.4560.objecter ms_dispatch 0x7f9808000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:23.805 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.803+0000 7f98057fa640 3 client.4560.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:23.805 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.803+0000 7f98057fa640 3 client.4560.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:23.805 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.803+0000 7f98057fa640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-25T18:20:23.805 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.803+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98000504f0 con 0x7f9808149a40 2024-03-25T18:20:23.805 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.803+0000 7f98057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:23.805 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:23.803+0000 7f98057fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:24.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.061+0000 7f980f063640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:24.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.061+0000 7f980f063640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:24.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.061+0000 7f980f063640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:20:24.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.061+0000 7f980f063640 1 -- 172.21.15.63:0/3243089904 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f980813db50 con 0x7f9808149a40 2024-03-25T18:20:24.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.083+0000 7f98057fa640 1 -- 172.21.15.63:0/3243089904 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f9800052420 con 0x7f9808149a40 2024-03-25T18:20:24.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.083+0000 7f98057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:24.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.083+0000 7f98057fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:24.084 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:20:24.084 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:24.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.085+0000 7f980f063640 10 librados: watch_flush enter 2024-03-25T18:20:24.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.086+0000 7f980f063640 10 librados: watch_flush exit 2024-03-25T18:20:24.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.086+0000 7f980f063640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-25T18:20:24.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.086+0000 7f980f063640 10 client.4560.objecter successfully canceled tick 2024-03-25T18:20:24.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.086+0000 7f980f063640 1 -- 172.21.15.63:0/3243089904 >> v1:172.21.15.81:6800/3640340873 conn(0x7f97ec06df60 legacy=0x7f97ec070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:24.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.086+0000 7f980f063640 10 monclient: shutdown 2024-03-25T18:20:24.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.086+0000 7f980f063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:24.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.086+0000 7f980f063640 1 -- 172.21.15.63:0/3243089904 >> v1:172.21.15.63:6789/0 conn(0x7f9808149a40 legacy=0x7f9808139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:24.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.086+0000 7f980f063640 1 -- 172.21.15.63:0/3243089904 shutdown_connections 2024-03-25T18:20:24.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.086+0000 7f980f063640 1 -- 172.21.15.63:0/3243089904 >> 172.21.15.63:0/3243089904 conn(0x7f98080b5c00 msgr2=0x7f98080b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:24.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.086+0000 7f980f063640 1 -- 172.21.15.63:0/3243089904 shutdown_connections 2024-03-25T18:20:24.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.086+0000 7f980f063640 1 -- 172.21.15.63:0/3243089904 wait complete. 2024-03-25T18:20:24.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:24.086+0000 7f980f063640 1 librados: shutdown 2024-03-25T18:20:24.100 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:30.907 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:31.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.022+0000 7fea4c123640 10 monclient: get_monmap_and_config 2024-03-25T18:20:31.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.022+0000 7fea4c123640 10 monclient: build_initial_monmap 2024-03-25T18:20:31.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.022+0000 7fea4c123640 10 monclient: monmap: 2024-03-25T18:20:31.023 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:20:31.023 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:31.023 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:20:31.023444+0000 2024-03-25T18:20:31.023 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:20:31.023444+0000 2024-03-25T18:20:31.024 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:31.024 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:31.024 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:20:31.024 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:20:31.024 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:20:31.024 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:31.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.023+0000 7fea4c123640 1 Processor -- start 2024-03-25T18:20:31.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.023+0000 7fea4c123640 1 -- start start 2024-03-25T18:20:31.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.023+0000 7fea4c123640 10 monclient: init 2024-03-25T18:20:31.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.024+0000 7fea4c123640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:31.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.024+0000 7fea4c123640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:31.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.024+0000 7fea4c123640 10 monclient(hunting): picked mon.noname-c con 0x7fea4406ea40 addr v1:172.21.15.81:6789/0 2024-03-25T18:20:31.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.024+0000 7fea4c123640 10 monclient(hunting): picked mon.noname-b con 0x7fea44071800 addr v1:172.21.15.63:6790/0 2024-03-25T18:20:31.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.024+0000 7fea4c123640 10 monclient(hunting): picked mon.noname-a con 0x7fea44071c40 addr v1:172.21.15.63:6789/0 2024-03-25T18:20:31.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.024+0000 7fea4c123640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4406f1f0 con 0x7fea44071c40 2024-03-25T18:20:31.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.024+0000 7fea4c123640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea44070d00 con 0x7fea4406ea40 2024-03-25T18:20:31.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.024+0000 7fea4c123640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea440720b0 con 0x7fea44071800 2024-03-25T18:20:31.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.024+0000 7fea4c123640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:31.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.024+0000 7fea4c123640 10 monclient(hunting): authenticate will time out at 7925.210603s 2024-03-25T18:20:31.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.028+0000 7fea49ebf640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fea4406ea40 0x7fea4413ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:42716/0 (socket says 172.21.15.63:42716) 2024-03-25T18:20:31.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.028+0000 7fea49ebf640 1 -- 172.21.15.63:0/2997860321 learned_addr learned my addr 172.21.15.63:0/2997860321 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:20:31.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.037+0000 7fea48ebd640 1 -- 172.21.15.63:0/2997860321 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771370490 0 0) 0x7fea44070d00 con 0x7fea4406ea40 2024-03-25T18:20:31.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.037+0000 7fea48ebd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:31.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.037+0000 7fea48ebd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:31.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.037+0000 7fea48ebd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:31.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.037+0000 7fea48ebd640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:20:31.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.037+0000 7fea48ebd640 1 -- 172.21.15.63:0/2997860321 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea30003520 con 0x7fea4406ea40 2024-03-25T18:20:31.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea48ebd640 1 -- 172.21.15.63:0/2997860321 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2872676282 0 0) 0x7fea30003520 con 0x7fea4406ea40 2024-03-25T18:20:31.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea48ebd640 1 -- 172.21.15.63:0/2997860321 >> v1:172.21.15.63:6790/0 conn(0x7fea44071800 legacy=0x7fea44147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:31.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea48ebd640 1 -- 172.21.15.63:0/2997860321 >> v1:172.21.15.63:6789/0 conn(0x7fea44071c40 legacy=0x7fea4414a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:31.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea48ebd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:31.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea48ebd640 1 monclient: found mon.noname-c 2024-03-25T18:20:31.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea48ebd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:31.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea48ebd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:20:31.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea48ebd640 1 -- 172.21.15.63:0/2997860321 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea44072780 con 0x7fea4406ea40 2024-03-25T18:20:31.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea48ebd640 10 monclient: _finish_auth 0 2024-03-25T18:20:31.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea48ebd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:31.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea48ebd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:31.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea48ebd640 1 -- 172.21.15.63:0/2997860321 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fea40001a10 con 0x7fea4406ea40 2024-03-25T18:20:31.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea48ebd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:31.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea48ebd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:31.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea48ebd640 10 monclient: dump: 2024-03-25T18:20:31.053 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:31.053 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:31.053 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:31.053 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:31.053 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:31.053 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:31.053 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:31.053 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:31.053 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:31.053 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:31.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea4c123640 5 monclient: authenticate success, global_id 4591 2024-03-25T18:20:31.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.050+0000 7fea4c123640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:31.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.055+0000 7fea48ebd640 1 -- 172.21.15.63:0/2997860321 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea40001b70 con 0x7fea4406ea40 2024-03-25T18:20:31.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.055+0000 7fea48ebd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:31.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.055+0000 7fea48ebd640 1 -- 172.21.15.63:0/2997860321 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fea40001f80 con 0x7fea4406ea40 2024-03-25T18:20:31.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.055+0000 7fea4c123640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:31.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.055+0000 7fea4c123640 4 set_mon_vals no callback set 2024-03-25T18:20:31.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.055+0000 7fea48ebd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:31.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.055+0000 7fea48ebd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:31.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.055+0000 7fea48ebd640 10 monclient: dump: 2024-03-25T18:20:31.057 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:31.057 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:31.057 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:31.057 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:31.057 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:31.057 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:31.057 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:31.057 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:31.057 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:31.057 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:31.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.056+0000 7fea4c123640 10 monclient: shutdown 2024-03-25T18:20:31.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.056+0000 7fea4c123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:31.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.056+0000 7fea4c123640 1 -- 172.21.15.63:0/2997860321 >> v1:172.21.15.81:6789/0 conn(0x7fea4406ea40 legacy=0x7fea4413ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:31.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.056+0000 7fea4c123640 1 -- 172.21.15.63:0/2997860321 shutdown_connections 2024-03-25T18:20:31.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.056+0000 7fea4c123640 1 -- 172.21.15.63:0/2997860321 >> 172.21.15.63:0/2997860321 conn(0x7fea4406b630 msgr2=0x7fea440b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:31.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.056+0000 7fea4c123640 1 -- 172.21.15.63:0/2997860321 shutdown_connections 2024-03-25T18:20:31.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.056+0000 7fea4c123640 1 -- 172.21.15.63:0/2997860321 wait complete. 2024-03-25T18:20:31.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.057+0000 7fea4c123640 10 monclient: build_initial_monmap 2024-03-25T18:20:31.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.057+0000 7fea4c123640 10 monclient: monmap: 2024-03-25T18:20:31.060 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:20:31.060 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:31.060 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:20:31.060 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:20:31.060 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:31.060 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:20:31.060 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:20:31.060 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:20:31.061 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:20:31.061 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:31.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.057+0000 7fea4c123640 1 librados: starting msgr at 2024-03-25T18:20:31.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.057+0000 7fea4c123640 1 librados: starting objecter 2024-03-25T18:20:31.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.057+0000 7fea4c123640 1 Processor -- start 2024-03-25T18:20:31.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.057+0000 7fea4c123640 1 -- start start 2024-03-25T18:20:31.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.057+0000 7fea4c123640 1 librados: setting wanted keys 2024-03-25T18:20:31.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.057+0000 7fea4c123640 1 librados: calling monclient init 2024-03-25T18:20:31.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.057+0000 7fea4c123640 10 monclient: init 2024-03-25T18:20:31.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.058+0000 7fea4c123640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:31.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.058+0000 7fea4c123640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:31.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.058+0000 7fea4c123640 10 monclient(hunting): picked mon.noname-a con 0x7fea4406ea40 addr v1:172.21.15.63:6789/0 2024-03-25T18:20:31.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.058+0000 7fea4c123640 10 monclient(hunting): picked mon.noname-b con 0x7fea44071800 addr v1:172.21.15.81:6789/0 2024-03-25T18:20:31.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.058+0000 7fea4c123640 10 monclient(hunting): picked mon.noname-c con 0x7fea44071c40 addr v1:172.21.15.63:6790/0 2024-03-25T18:20:31.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.058+0000 7fea4c123640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4413fb10 con 0x7fea4406ea40 2024-03-25T18:20:31.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.058+0000 7fea4c123640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea441323a0 con 0x7fea44071800 2024-03-25T18:20:31.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.058+0000 7fea4c123640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea441dbd80 con 0x7fea44071c40 2024-03-25T18:20:31.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.058+0000 7fea4c123640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:31.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.058+0000 7fea4c123640 10 monclient(hunting): authenticate will time out at 7925.244799s 2024-03-25T18:20:31.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.062+0000 7fea496be640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fea44071800 0x7fea441daf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:42746/0 (socket says 172.21.15.63:42746) 2024-03-25T18:20:31.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.062+0000 7fea496be640 1 -- 172.21.15.63:0/1455235576 learned_addr learned my addr 172.21.15.63:0/1455235576 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:20:31.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.063+0000 7fea3a7fc640 10 client.?.objecter ms_handle_connect 0x7fea44071800 2024-03-25T18:20:31.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.063+0000 7fea3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:31.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.065+0000 7fea3a7fc640 10 client.?.objecter ms_handle_connect 0x7fea44071c40 2024-03-25T18:20:31.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.065+0000 7fea3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:31.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.073+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163878098 0 0) 0x7fea441323a0 con 0x7fea44071800 2024-03-25T18:20:31.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.073+0000 7fea3a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:31.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.073+0000 7fea3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:31.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.073+0000 7fea3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:31.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.073+0000 7fea3a7fc640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:20:31.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.073+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea28004eb0 con 0x7fea44071800 2024-03-25T18:20:31.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.075+0000 7fea3a7fc640 10 client.?.objecter ms_handle_connect 0x7fea4406ea40 2024-03-25T18:20:31.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.075+0000 7fea3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:31.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.083+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791948469 0 0) 0x7fea441dbd80 con 0x7fea44071c40 2024-03-25T18:20:31.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.083+0000 7fea3a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:31.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.083+0000 7fea3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:31.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.083+0000 7fea3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:31.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.083+0000 7fea3a7fc640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:20:31.085 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.083+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea441323a0 con 0x7fea44071c40 2024-03-25T18:20:31.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.087+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2948883452 0 0) 0x7fea28004eb0 con 0x7fea44071800 2024-03-25T18:20:31.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.087+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea441dbd80 con 0x7fea44071800 2024-03-25T18:20:31.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.087+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fea34001a30 con 0x7fea44071800 2024-03-25T18:20:31.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.087+0000 7fea3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:31.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.087+0000 7fea3a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:31.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.087+0000 7fea3a7fc640 10 monclient(hunting): dump: 2024-03-25T18:20:31.089 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:31.089 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:31.089 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:31.089 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:31.089 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:31.089 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:31.089 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:31.089 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:31.090 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:31.090 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:31.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.087+0000 7fea3a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:31.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.096+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291458553 0 0) 0x7fea4413fb10 con 0x7fea4406ea40 2024-03-25T18:20:31.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.097+0000 7fea3a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:31.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.097+0000 7fea3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:31.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.097+0000 7fea3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:31.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.097+0000 7fea3a7fc640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:20:31.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.097+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea28004eb0 con 0x7fea4406ea40 2024-03-25T18:20:31.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2479004347 0 0) 0x7fea441dbd80 con 0x7fea44071800 2024-03-25T18:20:31.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 >> v1:172.21.15.63:6790/0 conn(0x7fea44071c40 legacy=0x7fea441db670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:31.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 >> v1:172.21.15.63:6789/0 conn(0x7fea4406ea40 legacy=0x7fea44148c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:31.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea3a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:31.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea3a7fc640 1 monclient: found mon.b 2024-03-25T18:20:31.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:31.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea3a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:31.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea44132610 con 0x7fea44071800 2024-03-25T18:20:31.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea3a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:31.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea3a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:31.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:31.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea4c123640 5 monclient: authenticate success, global_id 4594 2024-03-25T18:20:31.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea4c123640 10 monclient: _renew_subs 2024-03-25T18:20:31.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea4c123640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:31.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea4c123640 1 -- 172.21.15.63:0/1455235576 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea441328f0 con 0x7fea44071800 2024-03-25T18:20:31.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea4c123640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:31.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea4c123640 10 monclient: _renew_subs 2024-03-25T18:20:31.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea4c123640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:31.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea4c123640 1 -- 172.21.15.63:0/1455235576 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea44130f20 con 0x7fea44071800 2024-03-25T18:20:31.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.099+0000 7fea4c123640 1 librados: init done 2024-03-25T18:20:31.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.101+0000 7fea4c123640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:31.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.101+0000 7fea4c123640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:31.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.101+0000 7fea4c123640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:31.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.101+0000 7fea4c123640 1 -- 172.21.15.63:0/1455235576 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea440627b0 con 0x7fea44071800 2024-03-25T18:20:31.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.376+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea34001c70 con 0x7fea44071800 2024-03-25T18:20:31.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.376+0000 7fea3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:31.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.376+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fea34003a60 con 0x7fea44071800 2024-03-25T18:20:31.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.376+0000 7fea3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:31.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.376+0000 7fea3a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:31.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.376+0000 7fea3a7fc640 10 monclient: dump: 2024-03-25T18:20:31.378 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:31.378 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:31.378 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:31.378 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:31.378 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:31.378 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:31.378 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:31.378 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:31.378 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:31.378 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:31.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.376+0000 7fea3bfff640 4 set_mon_vals no callback set 2024-03-25T18:20:31.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.377+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fea34003c80 con 0x7fea44071800 2024-03-25T18:20:31.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.377+0000 7fea49ebf640 10 monclient: get_auth_request con 0x7fea2806de30 auth_method 0 2024-03-25T18:20:31.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.378+0000 7fea3a7fc640 10 client.4594.objecter ms_handle_connect 0x7fea2806de30 2024-03-25T18:20:31.382 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.381+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fea34054850 con 0x7fea44071800 2024-03-25T18:20:31.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.381+0000 7fea3a7fc640 10 client.4594.objecter ms_dispatch 0x7fea44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:31.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.381+0000 7fea3a7fc640 3 client.4594.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:31.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.381+0000 7fea3a7fc640 3 client.4594.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:31.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.381+0000 7fea3a7fc640 20 client.4594.objecter dump_active .. 0 homeless 2024-03-25T18:20:31.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.382+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea34050360 con 0x7fea44071800 2024-03-25T18:20:31.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.382+0000 7fea3a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:31.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.382+0000 7fea3a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:31.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.644+0000 7fea4c123640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:31.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.644+0000 7fea4c123640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:31.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.644+0000 7fea4c123640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:31.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.644+0000 7fea4c123640 1 -- 172.21.15.63:0/1455235576 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea440681e0 con 0x7fea44071800 2024-03-25T18:20:31.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.668+0000 7fea3a7fc640 1 -- 172.21.15.63:0/1455235576 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fea34052290 con 0x7fea44071800 2024-03-25T18:20:31.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.668+0000 7fea3a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:31.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.668+0000 7fea3a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:31.670 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:20:31.670 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:31.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.671+0000 7fea4c123640 10 librados: watch_flush enter 2024-03-25T18:20:31.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.671+0000 7fea4c123640 10 librados: watch_flush exit 2024-03-25T18:20:31.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.671+0000 7fea4c123640 20 client.4594.objecter shutdown clearing up homeless session... 2024-03-25T18:20:31.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.671+0000 7fea4c123640 10 client.4594.objecter successfully canceled tick 2024-03-25T18:20:31.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.671+0000 7fea4c123640 1 -- 172.21.15.63:0/1455235576 >> v1:172.21.15.81:6800/3640340873 conn(0x7fea2806de30 legacy=0x7fea280702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:31.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.671+0000 7fea4c123640 10 monclient: shutdown 2024-03-25T18:20:31.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.671+0000 7fea4c123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:31.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.671+0000 7fea4c123640 1 -- 172.21.15.63:0/1455235576 >> v1:172.21.15.81:6789/0 conn(0x7fea44071800 legacy=0x7fea441daf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:31.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.671+0000 7fea4c123640 1 -- 172.21.15.63:0/1455235576 shutdown_connections 2024-03-25T18:20:31.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.671+0000 7fea4c123640 1 -- 172.21.15.63:0/1455235576 >> 172.21.15.63:0/1455235576 conn(0x7fea4406b630 msgr2=0x7fea440af8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:31.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.671+0000 7fea4c123640 1 -- 172.21.15.63:0/1455235576 shutdown_connections 2024-03-25T18:20:31.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.671+0000 7fea4c123640 1 -- 172.21.15.63:0/1455235576 wait complete. 2024-03-25T18:20:31.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:31.672+0000 7fea4c123640 1 librados: shutdown 2024-03-25T18:20:31.685 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:38.491 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:38.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.597+0000 7ffbe213d640 10 monclient: get_monmap_and_config 2024-03-25T18:20:38.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.597+0000 7ffbe213d640 10 monclient: build_initial_monmap 2024-03-25T18:20:38.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.597+0000 7ffbe213d640 10 monclient: monmap: 2024-03-25T18:20:38.598 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:20:38.599 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:38.599 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:20:38.598645+0000 2024-03-25T18:20:38.599 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:20:38.598645+0000 2024-03-25T18:20:38.599 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:38.599 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:38.599 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:20:38.599 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:20:38.599 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:20:38.599 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:38.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.598+0000 7ffbe213d640 1 Processor -- start 2024-03-25T18:20:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.598+0000 7ffbe213d640 1 -- start start 2024-03-25T18:20:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.598+0000 7ffbe213d640 10 monclient: init 2024-03-25T18:20:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.599+0000 7ffbe213d640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.599+0000 7ffbe213d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.599+0000 7ffbe213d640 10 monclient(hunting): picked mon.noname-b con 0x7ffbdc0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:20:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.599+0000 7ffbe213d640 10 monclient(hunting): picked mon.noname-a con 0x7ffbdc13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:20:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.599+0000 7ffbe213d640 10 monclient(hunting): picked mon.noname-c con 0x7ffbdc1430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:20:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.599+0000 7ffbe213d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbdc142520 con 0x7ffbdc13f710 2024-03-25T18:20:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.599+0000 7ffbe213d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbdc0b28a0 con 0x7ffbdc1430c0 2024-03-25T18:20:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.599+0000 7ffbe213d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbdc0b2a30 con 0x7ffbdc0a98f0 2024-03-25T18:20:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.599+0000 7ffbe213d640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.599+0000 7ffbe213d640 10 monclient(hunting): authenticate will time out at 7932.785642s 2024-03-25T18:20:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.599+0000 7ffbdb7fe640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7ffbdc0a98f0 0x7ffbdc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:55626/0 (socket says 172.21.15.63:55626) 2024-03-25T18:20:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.600+0000 7ffbdb7fe640 1 -- 172.21.15.63:0/3928181132 learned_addr learned my addr 172.21.15.63:0/3928181132 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:20:38.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.604+0000 7ffbda7fc640 1 -- 172.21.15.63:0/3928181132 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574896021 0 0) 0x7ffbdc0b28a0 con 0x7ffbdc1430c0 2024-03-25T18:20:38.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.604+0000 7ffbda7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:38.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.604+0000 7ffbda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:38.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.604+0000 7ffbda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:38.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.604+0000 7ffbda7fc640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:20:38.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.604+0000 7ffbda7fc640 1 -- 172.21.15.63:0/3928181132 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbc4003520 con 0x7ffbdc1430c0 2024-03-25T18:20:38.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbda7fc640 1 -- 172.21.15.63:0/3928181132 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1210801119 0 0) 0x7ffbc4003520 con 0x7ffbdc1430c0 2024-03-25T18:20:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbda7fc640 1 -- 172.21.15.63:0/3928181132 >> v1:172.21.15.63:6790/0 conn(0x7ffbdc0a98f0 legacy=0x7ffbdc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbda7fc640 1 -- 172.21.15.63:0/3928181132 >> v1:172.21.15.63:6789/0 conn(0x7ffbdc13f710 legacy=0x7ffbdc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:20:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbda7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbda7fc640 1 monclient: found mon.noname-c 2024-03-25T18:20:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbda7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:20:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbda7fc640 1 -- 172.21.15.63:0/3928181132 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbdc0b44c0 con 0x7ffbdc1430c0 2024-03-25T18:20:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbda7fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbda7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbda7fc640 1 -- 172.21.15.63:0/3928181132 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ffbd4001a10 con 0x7ffbdc1430c0 2024-03-25T18:20:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbda7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbda7fc640 10 monclient: dump: 2024-03-25T18:20:38.617 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:38.617 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:38.617 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:38.617 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:38.617 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:38.617 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:38.617 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:38.617 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:38.617 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:38.617 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:38.617 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbe213d640 5 monclient: authenticate success, global_id 4597 2024-03-25T18:20:38.617 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.614+0000 7ffbe213d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:38.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.617+0000 7ffbda7fc640 1 -- 172.21.15.63:0/3928181132 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbd4001be0 con 0x7ffbdc1430c0 2024-03-25T18:20:38.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.617+0000 7ffbda7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:38.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.617+0000 7ffbda7fc640 1 -- 172.21.15.63:0/3928181132 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ffbd4002030 con 0x7ffbdc1430c0 2024-03-25T18:20:38.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.617+0000 7ffbe213d640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:38.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.617+0000 7ffbe213d640 4 set_mon_vals no callback set 2024-03-25T18:20:38.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.617+0000 7ffbda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:38.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.617+0000 7ffbda7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:38.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.617+0000 7ffbda7fc640 10 monclient: dump: 2024-03-25T18:20:38.619 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:38.619 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:38.619 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:38.619 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:38.619 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:38.619 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:38.619 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:38.619 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:38.619 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:38.619 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:38.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.618+0000 7ffbe213d640 10 monclient: shutdown 2024-03-25T18:20:38.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.618+0000 7ffbe213d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:38.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.618+0000 7ffbe213d640 1 -- 172.21.15.63:0/3928181132 >> v1:172.21.15.81:6789/0 conn(0x7ffbdc1430c0 legacy=0x7ffbdc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:38.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.618+0000 7ffbe213d640 1 -- 172.21.15.63:0/3928181132 shutdown_connections 2024-03-25T18:20:38.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.618+0000 7ffbe213d640 1 -- 172.21.15.63:0/3928181132 >> 172.21.15.63:0/3928181132 conn(0x7ffbdc0b7ea0 msgr2=0x7ffbdc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:38.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.618+0000 7ffbe213d640 1 -- 172.21.15.63:0/3928181132 shutdown_connections 2024-03-25T18:20:38.622 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.618+0000 7ffbe213d640 1 -- 172.21.15.63:0/3928181132 wait complete. 2024-03-25T18:20:38.622 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.619+0000 7ffbe213d640 10 monclient: build_initial_monmap 2024-03-25T18:20:38.622 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.619+0000 7ffbe213d640 10 monclient: monmap: 2024-03-25T18:20:38.622 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:20:38.622 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:38.622 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:20:38.622 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:20:38.622 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:38.622 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:20:38.622 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:20:38.622 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:20:38.622 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:20:38.622 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:38.622 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.619+0000 7ffbe213d640 1 librados: starting msgr at 2024-03-25T18:20:38.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.619+0000 7ffbe213d640 1 librados: starting objecter 2024-03-25T18:20:38.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.619+0000 7ffbe213d640 1 Processor -- start 2024-03-25T18:20:38.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.619+0000 7ffbe213d640 1 -- start start 2024-03-25T18:20:38.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.619+0000 7ffbe213d640 1 librados: setting wanted keys 2024-03-25T18:20:38.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.619+0000 7ffbe213d640 1 librados: calling monclient init 2024-03-25T18:20:38.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.619+0000 7ffbe213d640 10 monclient: init 2024-03-25T18:20:38.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.620+0000 7ffbe213d640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:38.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.620+0000 7ffbe213d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:38.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.620+0000 7ffbe213d640 10 monclient(hunting): picked mon.noname-c con 0x7ffbdc0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:20:38.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.620+0000 7ffbe213d640 10 monclient(hunting): picked mon.noname-b con 0x7ffbdc13f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:20:38.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.620+0000 7ffbe213d640 10 monclient(hunting): picked mon.noname-a con 0x7ffbdc1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:20:38.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.620+0000 7ffbe213d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbdc1d8410 con 0x7ffbdc1430c0 2024-03-25T18:20:38.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.620+0000 7ffbe213d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbdc0abda0 con 0x7ffbdc13f710 2024-03-25T18:20:38.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.620+0000 7ffbe213d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbdc1d7500 con 0x7ffbdc0a98f0 2024-03-25T18:20:38.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.620+0000 7ffbe213d640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:38.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.620+0000 7ffbe213d640 10 monclient(hunting): authenticate will time out at 7932.806784s 2024-03-25T18:20:38.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.628+0000 7ffbdbfff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7ffbdc1430c0 0x7ffbdc133f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48874/0 (socket says 172.21.15.63:48874) 2024-03-25T18:20:38.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.628+0000 7ffbdbfff640 1 -- 172.21.15.63:0/2806408591 learned_addr learned my addr 172.21.15.63:0/2806408591 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:20:38.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.632+0000 7ffbc3fff640 10 client.?.objecter ms_handle_connect 0x7ffbdc1430c0 2024-03-25T18:20:38.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.632+0000 7ffbc3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:38.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.639+0000 7ffbc3fff640 10 client.?.objecter ms_handle_connect 0x7ffbdc13f710 2024-03-25T18:20:38.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.639+0000 7ffbc3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:38.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.642+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271111741 0 0) 0x7ffbdc0abda0 con 0x7ffbdc13f710 2024-03-25T18:20:38.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.642+0000 7ffbc3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:38.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.642+0000 7ffbc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:38.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.642+0000 7ffbc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:38.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.642+0000 7ffbc3fff640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:20:38.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.642+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbc8004eb0 con 0x7ffbdc13f710 2024-03-25T18:20:38.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.644+0000 7ffbc3fff640 10 client.?.objecter ms_handle_connect 0x7ffbdc0a98f0 2024-03-25T18:20:38.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.644+0000 7ffbc3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:38.648 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.647+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517136577 0 0) 0x7ffbdc1d8410 con 0x7ffbdc1430c0 2024-03-25T18:20:38.648 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.647+0000 7ffbc3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:38.648 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.647+0000 7ffbc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:38.648 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.647+0000 7ffbc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:38.648 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.647+0000 7ffbc3fff640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:20:38.648 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.647+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbdc0abda0 con 0x7ffbdc1430c0 2024-03-25T18:20:38.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.650+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170203912 0 0) 0x7ffbdc1d7500 con 0x7ffbdc0a98f0 2024-03-25T18:20:38.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.650+0000 7ffbc3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:38.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.650+0000 7ffbc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:38.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.650+0000 7ffbc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:38.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.650+0000 7ffbc3fff640 10 monclient(hunting): my global_id is 4574 2024-03-25T18:20:38.652 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.650+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbdc1d8410 con 0x7ffbdc0a98f0 2024-03-25T18:20:38.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.654+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4285843854 0 0) 0x7ffbc8004eb0 con 0x7ffbdc13f710 2024-03-25T18:20:38.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.655+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbdc1d7500 con 0x7ffbdc13f710 2024-03-25T18:20:38.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.655+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ffbd0001870 con 0x7ffbdc13f710 2024-03-25T18:20:38.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.655+0000 7ffbc3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:38.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.655+0000 7ffbc3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:38.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.655+0000 7ffbc3fff640 10 monclient(hunting): dump: 2024-03-25T18:20:38.656 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:38.656 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:38.657 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:38.657 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:38.657 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:38.657 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:38.657 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:38.657 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:38.657 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:38.657 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:38.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.655+0000 7ffbc3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:38.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 7937305 0 0) 0x7ffbdc1d7500 con 0x7ffbdc13f710 2024-03-25T18:20:38.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 >> v1:172.21.15.63:6790/0 conn(0x7ffbdc0a98f0 legacy=0x7ffbdc137c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:38.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 >> v1:172.21.15.63:6789/0 conn(0x7ffbdc1430c0 legacy=0x7ffbdc133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:38.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbc3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:38.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbc3fff640 1 monclient: found mon.b 2024-03-25T18:20:38.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:38.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbc3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:38.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbdc1d77e0 con 0x7ffbdc13f710 2024-03-25T18:20:38.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbc3fff640 10 monclient: _finish_auth 0 2024-03-25T18:20:38.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbc3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:38.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:38.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbe213d640 5 monclient: authenticate success, global_id 4600 2024-03-25T18:20:38.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbe213d640 10 monclient: _renew_subs 2024-03-25T18:20:38.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbe213d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:38.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbe213d640 1 -- 172.21.15.63:0/2806408591 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbdc138540 con 0x7ffbdc13f710 2024-03-25T18:20:38.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbe213d640 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:38.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbe213d640 10 monclient: _renew_subs 2024-03-25T18:20:38.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbe213d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:38.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbe213d640 1 -- 172.21.15.63:0/2806408591 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbdc134690 con 0x7ffbdc13f710 2024-03-25T18:20:38.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.661+0000 7ffbe213d640 1 librados: init done 2024-03-25T18:20:38.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.662+0000 7ffbe213d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:38.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.662+0000 7ffbe213d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:38.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.662+0000 7ffbe213d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:38.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.662+0000 7ffbe213d640 1 -- 172.21.15.63:0/2806408591 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbdc0b3ef0 con 0x7ffbdc13f710 2024-03-25T18:20:38.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.671+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbd00019d0 con 0x7ffbdc13f710 2024-03-25T18:20:38.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.671+0000 7ffbc3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:38.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.671+0000 7ffbd9ffb640 4 set_mon_vals no callback set 2024-03-25T18:20:38.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.948+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7ffbd00037d0 con 0x7ffbdc13f710 2024-03-25T18:20:38.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.948+0000 7ffbc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:38.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.948+0000 7ffbc3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:38.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.948+0000 7ffbc3fff640 10 monclient: dump: 2024-03-25T18:20:38.949 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:38.949 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:38.949 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:38.949 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:38.949 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:38.949 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:38.949 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:38.950 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:38.950 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:38.950 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:38.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.949+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7ffbd0019820 con 0x7ffbdc13f710 2024-03-25T18:20:38.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.950+0000 7ffbdb7fe640 10 monclient: get_auth_request con 0x7ffbc806dda0 auth_method 0 2024-03-25T18:20:38.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.950+0000 7ffbc3fff640 10 client.4600.objecter ms_handle_connect 0x7ffbc806dda0 2024-03-25T18:20:38.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.951+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7ffbd00535e0 con 0x7ffbdc13f710 2024-03-25T18:20:38.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.951+0000 7ffbc3fff640 10 client.4600.objecter ms_dispatch 0x7ffbdc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:38.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.951+0000 7ffbc3fff640 3 client.4600.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:38.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.951+0000 7ffbc3fff640 3 client.4600.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:38.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.951+0000 7ffbc3fff640 20 client.4600.objecter dump_active .. 0 homeless 2024-03-25T18:20:38.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.955+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffbd0082e60 con 0x7ffbdc13f710 2024-03-25T18:20:38.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.955+0000 7ffbc3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:38.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:38.955+0000 7ffbc3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:39.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.211+0000 7ffbe213d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:39.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.211+0000 7ffbe213d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:39.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.211+0000 7ffbe213d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:39.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.211+0000 7ffbe213d640 1 -- 172.21.15.63:0/2806408591 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbdc1d7970 con 0x7ffbdc13f710 2024-03-25T18:20:39.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.232+0000 7ffbc3fff640 1 -- 172.21.15.63:0/2806408591 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7ffbd0052cb0 con 0x7ffbdc13f710 2024-03-25T18:20:39.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.232+0000 7ffbc3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:39.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.232+0000 7ffbc3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:39.233 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:20:39.233 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:39.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.234+0000 7ffbe213d640 10 librados: watch_flush enter 2024-03-25T18:20:39.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.234+0000 7ffbe213d640 10 librados: watch_flush exit 2024-03-25T18:20:39.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.234+0000 7ffbe213d640 20 client.4600.objecter shutdown clearing up homeless session... 2024-03-25T18:20:39.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.234+0000 7ffbe213d640 10 client.4600.objecter successfully canceled tick 2024-03-25T18:20:39.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.234+0000 7ffbe213d640 1 -- 172.21.15.63:0/2806408591 >> v1:172.21.15.81:6800/3640340873 conn(0x7ffbc806dda0 legacy=0x7ffbc8070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:39.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.234+0000 7ffbe213d640 10 monclient: shutdown 2024-03-25T18:20:39.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.234+0000 7ffbe213d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:39.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.234+0000 7ffbe213d640 1 -- 172.21.15.63:0/2806408591 >> v1:172.21.15.81:6789/0 conn(0x7ffbdc13f710 legacy=0x7ffbdc1d7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:39.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.234+0000 7ffbe213d640 1 -- 172.21.15.63:0/2806408591 shutdown_connections 2024-03-25T18:20:39.237 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.234+0000 7ffbe213d640 1 -- 172.21.15.63:0/2806408591 >> 172.21.15.63:0/2806408591 conn(0x7ffbdc0b7ea0 msgr2=0x7ffbdc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:39.237 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.234+0000 7ffbe213d640 1 -- 172.21.15.63:0/2806408591 shutdown_connections 2024-03-25T18:20:39.237 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.234+0000 7ffbe213d640 1 -- 172.21.15.63:0/2806408591 wait complete. 2024-03-25T18:20:39.237 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:39.235+0000 7ffbe213d640 1 librados: shutdown 2024-03-25T18:20:39.248 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:44.596 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:26.288 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:44.612 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:26.304 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:44.648 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:26.340 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:44.650 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:26.342 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:44.665 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:26.357 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:44.671 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:08:26.332 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:44.680 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:08:26.340 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:44.683 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:26.375 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:44.705 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:08:26.365 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:46.055 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:46.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.166+0000 7f453259d640 10 monclient: get_monmap_and_config 2024-03-25T18:20:46.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.166+0000 7f453259d640 10 monclient: build_initial_monmap 2024-03-25T18:20:46.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.166+0000 7f453259d640 10 monclient: monmap: 2024-03-25T18:20:46.167 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:20:46.167 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:46.167 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:20:46.167331+0000 2024-03-25T18:20:46.167 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:20:46.167331+0000 2024-03-25T18:20:46.168 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:46.168 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:46.168 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:20:46.168 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:20:46.168 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:20:46.168 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:46.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.166+0000 7f453259d640 1 Processor -- start 2024-03-25T18:20:46.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.167+0000 7f453259d640 1 -- start start 2024-03-25T18:20:46.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.167+0000 7f453259d640 10 monclient: init 2024-03-25T18:20:46.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.167+0000 7f453259d640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:46.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.167+0000 7f453259d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:46.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.167+0000 7f453259d640 10 monclient(hunting): picked mon.noname-b con 0x7f452c0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:20:46.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.167+0000 7f453259d640 10 monclient(hunting): picked mon.noname-c con 0x7f452c13f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:20:46.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.167+0000 7f453259d640 10 monclient(hunting): picked mon.noname-a con 0x7f452c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:20:46.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.167+0000 7f453259d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452c142520 con 0x7f452c1430c0 2024-03-25T18:20:46.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.167+0000 7f453259d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452c0b28a0 con 0x7f452c13f710 2024-03-25T18:20:46.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.167+0000 7f453259d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452c0b2a30 con 0x7f452c0a98f0 2024-03-25T18:20:46.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.167+0000 7f453259d640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:46.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.167+0000 7f453259d640 10 monclient(hunting): authenticate will time out at 7940.353989s 2024-03-25T18:20:46.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.174+0000 7f452bfff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f452c0a98f0 0x7f452c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:60302/0 (socket says 172.21.15.63:60302) 2024-03-25T18:20:46.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.174+0000 7f452bfff640 1 -- 172.21.15.63:0/4006877504 learned_addr learned my addr 172.21.15.63:0/4006877504 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:20:46.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.185+0000 7f452affd640 1 -- 172.21.15.63:0/4006877504 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686840932 0 0) 0x7f452c0b2a30 con 0x7f452c0a98f0 2024-03-25T18:20:46.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.185+0000 7f452affd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:46.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.185+0000 7f452affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:46.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.185+0000 7f452affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:46.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.185+0000 7f452affd640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:20:46.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.185+0000 7f452affd640 1 -- 172.21.15.63:0/4006877504 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4514003520 con 0x7f452c0a98f0 2024-03-25T18:20:46.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.198+0000 7f452affd640 1 -- 172.21.15.63:0/4006877504 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 531026714 0 0) 0x7f4514003520 con 0x7f452c0a98f0 2024-03-25T18:20:46.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.198+0000 7f452affd640 1 -- 172.21.15.63:0/4006877504 >> v1:172.21.15.81:6789/0 conn(0x7f452c13f710 legacy=0x7f452c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:46.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.198+0000 7f452affd640 1 -- 172.21.15.63:0/4006877504 >> v1:172.21.15.63:6789/0 conn(0x7f452c1430c0 legacy=0x7f452c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:20:46.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.198+0000 7f452affd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:46.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.198+0000 7f452affd640 1 monclient: found mon.noname-b 2024-03-25T18:20:46.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.198+0000 7f452affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:46.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.198+0000 7f452affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:20:46.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.198+0000 7f452affd640 1 -- 172.21.15.63:0/4006877504 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452c0b44c0 con 0x7f452c0a98f0 2024-03-25T18:20:46.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.199+0000 7f452affd640 10 monclient: _finish_auth 0 2024-03-25T18:20:46.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.199+0000 7f452affd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:46.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.199+0000 7f452affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:46.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.199+0000 7f452affd640 1 -- 172.21.15.63:0/4006877504 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f451c001690 con 0x7f452c0a98f0 2024-03-25T18:20:46.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.199+0000 7f453259d640 5 monclient: authenticate success, global_id 4577 2024-03-25T18:20:46.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.199+0000 7f453259d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:46.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.199+0000 7f452affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:46.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.199+0000 7f452affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:46.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.199+0000 7f452affd640 10 monclient: dump: 2024-03-25T18:20:46.201 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:46.201 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:46.201 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:46.202 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:46.202 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:46.202 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:46.202 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:46.202 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:46.202 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:46.202 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:46.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.199+0000 7f453259d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:46.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.219+0000 7f452affd640 1 -- 172.21.15.63:0/4006877504 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f451c001860 con 0x7f452c0a98f0 2024-03-25T18:20:46.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.219+0000 7f452affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:46.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.219+0000 7f452affd640 1 -- 172.21.15.63:0/4006877504 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f451c001cb0 con 0x7f452c0a98f0 2024-03-25T18:20:46.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.219+0000 7f452affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:46.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.219+0000 7f452affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:46.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.219+0000 7f452affd640 10 monclient: dump: 2024-03-25T18:20:46.221 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:46.221 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:46.221 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:46.221 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:46.221 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:46.221 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:46.221 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:46.221 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:46.221 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:46.222 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:46.222 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.219+0000 7f453259d640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:46.222 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.219+0000 7f453259d640 4 set_mon_vals no callback set 2024-03-25T18:20:46.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.220+0000 7f453259d640 10 monclient: shutdown 2024-03-25T18:20:46.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.220+0000 7f453259d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:46.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.220+0000 7f453259d640 1 -- 172.21.15.63:0/4006877504 >> v1:172.21.15.63:6790/0 conn(0x7f452c0a98f0 legacy=0x7f452c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:46.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.221+0000 7f453259d640 1 -- 172.21.15.63:0/4006877504 shutdown_connections 2024-03-25T18:20:46.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.221+0000 7f453259d640 1 -- 172.21.15.63:0/4006877504 >> 172.21.15.63:0/4006877504 conn(0x7f452c0b7ea0 msgr2=0x7f452c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:46.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.221+0000 7f453259d640 1 -- 172.21.15.63:0/4006877504 shutdown_connections 2024-03-25T18:20:46.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.221+0000 7f453259d640 1 -- 172.21.15.63:0/4006877504 wait complete. 2024-03-25T18:20:46.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.222+0000 7f453259d640 10 monclient: build_initial_monmap 2024-03-25T18:20:46.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.222+0000 7f453259d640 10 monclient: monmap: 2024-03-25T18:20:46.223 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:20:46.223 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:46.223 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:20:46.223 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:20:46.224 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:46.224 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:20:46.224 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:20:46.224 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:20:46.224 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:20:46.224 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:46.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.222+0000 7f453259d640 1 librados: starting msgr at 2024-03-25T18:20:46.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.222+0000 7f453259d640 1 librados: starting objecter 2024-03-25T18:20:46.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.222+0000 7f453259d640 1 Processor -- start 2024-03-25T18:20:46.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.222+0000 7f453259d640 1 -- start start 2024-03-25T18:20:46.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.222+0000 7f453259d640 1 librados: setting wanted keys 2024-03-25T18:20:46.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.222+0000 7f453259d640 1 librados: calling monclient init 2024-03-25T18:20:46.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.222+0000 7f453259d640 10 monclient: init 2024-03-25T18:20:46.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.223+0000 7f453259d640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:46.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.223+0000 7f453259d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:46.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.223+0000 7f453259d640 10 monclient(hunting): picked mon.noname-a con 0x7f452c0a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:20:46.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.223+0000 7f453259d640 10 monclient(hunting): picked mon.noname-c con 0x7f452c13f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:20:46.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.223+0000 7f453259d640 10 monclient(hunting): picked mon.noname-b con 0x7f452c1430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:20:46.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.223+0000 7f453259d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452c1db4e0 con 0x7f452c0a98f0 2024-03-25T18:20:46.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.223+0000 7f453259d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452c14dcc0 con 0x7f452c1430c0 2024-03-25T18:20:46.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.223+0000 7f453259d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452c1db9c0 con 0x7f452c13f710 2024-03-25T18:20:46.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.223+0000 7f453259d640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:46.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.223+0000 7f453259d640 10 monclient(hunting): authenticate will time out at 7940.409791s 2024-03-25T18:20:46.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.225+0000 7f4530b3a640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f452c1430c0 0x7f452c14d5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:54400/0 (socket says 172.21.15.63:54400) 2024-03-25T18:20:46.227 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.225+0000 7f4530b3a640 1 -- 172.21.15.63:0/775473455 learned_addr learned my addr 172.21.15.63:0/775473455 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:20:46.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.228+0000 7f4528ff9640 10 client.?.objecter ms_handle_connect 0x7f452c1430c0 2024-03-25T18:20:46.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.228+0000 7f4528ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:46.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.233+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282640575 0 0) 0x7f452c14dcc0 con 0x7f452c1430c0 2024-03-25T18:20:46.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.233+0000 7f4528ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:46.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.233+0000 7f4528ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:46.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.233+0000 7f4528ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:46.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.233+0000 7f4528ff9640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:20:46.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.233+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4518004eb0 con 0x7f452c1430c0 2024-03-25T18:20:46.237 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.235+0000 7f4528ff9640 10 client.?.objecter ms_handle_connect 0x7f452c0a98f0 2024-03-25T18:20:46.237 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.235+0000 7f4528ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:46.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.241+0000 7f4528ff9640 10 client.?.objecter ms_handle_connect 0x7f452c13f710 2024-03-25T18:20:46.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.241+0000 7f4528ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:46.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.242+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1729270960 0 0) 0x7f4518004eb0 con 0x7f452c1430c0 2024-03-25T18:20:46.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.242+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f452c14dcc0 con 0x7f452c1430c0 2024-03-25T18:20:46.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.242+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f45240042e0 con 0x7f452c1430c0 2024-03-25T18:20:46.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.242+0000 7f4528ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:46.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.242+0000 7f4528ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:46.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.242+0000 7f4528ff9640 10 monclient(hunting): dump: 2024-03-25T18:20:46.244 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:46.244 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:46.244 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:46.244 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:46.244 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:46.244 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:46.244 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:46.244 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:46.244 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:46.244 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:46.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.242+0000 7f4528ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:46.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.243+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844841524 0 0) 0x7f452c1db4e0 con 0x7f452c0a98f0 2024-03-25T18:20:46.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.243+0000 7f4528ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:46.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.243+0000 7f4528ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:46.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.243+0000 7f4528ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:46.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.243+0000 7f4528ff9640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:20:46.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.243+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4518004eb0 con 0x7f452c0a98f0 2024-03-25T18:20:46.252 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.251+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599115854 0 0) 0x7f452c1db9c0 con 0x7f452c13f710 2024-03-25T18:20:46.252 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.251+0000 7f4528ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:46.252 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.251+0000 7f4528ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:46.252 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.251+0000 7f4528ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:46.252 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.251+0000 7f4528ff9640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:20:46.252 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.251+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f452c1db4e0 con 0x7f452c13f710 2024-03-25T18:20:46.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2890238921 0 0) 0x7f452c14dcc0 con 0x7f452c1430c0 2024-03-25T18:20:46.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 >> v1:172.21.15.63:6790/0 conn(0x7f452c13f710 legacy=0x7f452c1dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:46.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 >> v1:172.21.15.63:6789/0 conn(0x7f452c0a98f0 legacy=0x7f452c0b11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:46.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f4528ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:20:46.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f4528ff9640 1 monclient: found mon.b 2024-03-25T18:20:46.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f4528ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:46.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f4528ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:46.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452c14e4f0 con 0x7f452c1430c0 2024-03-25T18:20:46.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f4528ff9640 10 monclient: _finish_auth 0 2024-03-25T18:20:46.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f4528ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:20:46.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f4528ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:46.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f453259d640 5 monclient: authenticate success, global_id 4606 2024-03-25T18:20:46.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f453259d640 10 monclient: _renew_subs 2024-03-25T18:20:46.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f453259d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:46.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f453259d640 1 -- 172.21.15.63:0/775473455 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452c14e740 con 0x7f452c1430c0 2024-03-25T18:20:46.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f453259d640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:46.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f453259d640 10 monclient: _renew_subs 2024-03-25T18:20:46.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f453259d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:46.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f453259d640 1 -- 172.21.15.63:0/775473455 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f452c1d43c0 con 0x7f452c1430c0 2024-03-25T18:20:46.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.253+0000 7f453259d640 1 librados: init done 2024-03-25T18:20:46.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.254+0000 7f453259d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:46.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.254+0000 7f453259d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:46.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.254+0000 7f453259d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:46.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.254+0000 7f453259d640 1 -- 172.21.15.63:0/775473455 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f452c1db2f0 con 0x7f452c1430c0 2024-03-25T18:20:46.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.528+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4524004830 con 0x7f452c1430c0 2024-03-25T18:20:46.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.528+0000 7f4528ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:46.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.528+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f45240013f0 con 0x7f452c1430c0 2024-03-25T18:20:46.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.528+0000 7f4528ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:46.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.528+0000 7f4528ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:46.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.528+0000 7f4528ff9640 10 monclient: dump: 2024-03-25T18:20:46.530 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:46.530 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:46.530 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:46.530 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:46.530 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:46.530 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:46.531 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:46.531 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:46.531 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:46.531 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:46.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.528+0000 7f452a7fc640 4 set_mon_vals no callback set 2024-03-25T18:20:46.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.529+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f4524001610 con 0x7f452c1430c0 2024-03-25T18:20:46.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.530+0000 7f452bfff640 10 monclient: get_auth_request con 0x7f451806de60 auth_method 0 2024-03-25T18:20:46.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.530+0000 7f4528ff9640 10 client.4606.objecter ms_handle_connect 0x7f451806de60 2024-03-25T18:20:46.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.536+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f4524054a60 con 0x7f452c1430c0 2024-03-25T18:20:46.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.536+0000 7f4528ff9640 10 client.4606.objecter ms_dispatch 0x7f452c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:46.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.536+0000 7f4528ff9640 3 client.4606.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:46.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.536+0000 7f4528ff9640 3 client.4606.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:46.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.536+0000 7f4528ff9640 20 client.4606.objecter dump_active .. 0 homeless 2024-03-25T18:20:46.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.537+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f45240503a0 con 0x7f452c1430c0 2024-03-25T18:20:46.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.537+0000 7f4528ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:46.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.537+0000 7f4528ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:46.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.772+0000 7f453259d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:46.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.772+0000 7f453259d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:46.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.772+0000 7f453259d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:20:46.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.772+0000 7f453259d640 1 -- 172.21.15.63:0/775473455 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f452c14f2d0 con 0x7f452c1430c0 2024-03-25T18:20:46.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.794+0000 7f4528ff9640 1 -- 172.21.15.63:0/775473455 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f45240524a0 con 0x7f452c1430c0 2024-03-25T18:20:46.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.794+0000 7f4528ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:46.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.794+0000 7f4528ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:46.796 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:20:46.796 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:46.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.797+0000 7f453259d640 10 librados: watch_flush enter 2024-03-25T18:20:46.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.797+0000 7f453259d640 10 librados: watch_flush exit 2024-03-25T18:20:46.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.797+0000 7f453259d640 20 client.4606.objecter shutdown clearing up homeless session... 2024-03-25T18:20:46.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.797+0000 7f453259d640 10 client.4606.objecter successfully canceled tick 2024-03-25T18:20:46.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.797+0000 7f453259d640 1 -- 172.21.15.63:0/775473455 >> v1:172.21.15.81:6800/3640340873 conn(0x7f451806de60 legacy=0x7f4518070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:46.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.797+0000 7f453259d640 10 monclient: shutdown 2024-03-25T18:20:46.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.797+0000 7f453259d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:46.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.797+0000 7f453259d640 1 -- 172.21.15.63:0/775473455 >> v1:172.21.15.81:6789/0 conn(0x7f452c1430c0 legacy=0x7f452c14d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:46.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.797+0000 7f453259d640 1 -- 172.21.15.63:0/775473455 shutdown_connections 2024-03-25T18:20:46.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.797+0000 7f453259d640 1 -- 172.21.15.63:0/775473455 >> 172.21.15.63:0/775473455 conn(0x7f452c0b7ea0 msgr2=0x7f452c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:46.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.797+0000 7f453259d640 1 -- 172.21.15.63:0/775473455 shutdown_connections 2024-03-25T18:20:46.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.797+0000 7f453259d640 1 -- 172.21.15.63:0/775473455 wait complete. 2024-03-25T18:20:46.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:46.797+0000 7f453259d640 1 librados: shutdown 2024-03-25T18:20:46.809 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:53.615 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:53.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.729+0000 7f9e8c9af640 10 monclient: get_monmap_and_config 2024-03-25T18:20:53.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.729+0000 7f9e8c9af640 10 monclient: build_initial_monmap 2024-03-25T18:20:53.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.729+0000 7f9e8c9af640 10 monclient: monmap: 2024-03-25T18:20:53.731 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:20:53.731 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:53.731 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:20:53.730634+0000 2024-03-25T18:20:53.731 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:20:53.730634+0000 2024-03-25T18:20:53.731 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:53.731 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:53.731 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:20:53.731 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:20:53.731 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:20:53.731 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:53.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.730+0000 7f9e8c9af640 1 Processor -- start 2024-03-25T18:20:53.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.730+0000 7f9e8c9af640 1 -- start start 2024-03-25T18:20:53.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.730+0000 7f9e8c9af640 10 monclient: init 2024-03-25T18:20:53.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.730+0000 7f9e8c9af640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:53.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.730+0000 7f9e8c9af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:53.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.730+0000 7f9e8c9af640 10 monclient(hunting): picked mon.noname-b con 0x7f9e880a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:20:53.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.730+0000 7f9e8c9af640 10 monclient(hunting): picked mon.noname-c con 0x7f9e8813f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:20:53.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.731+0000 7f9e8c9af640 10 monclient(hunting): picked mon.noname-a con 0x7f9e881430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:20:53.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.731+0000 7f9e8c9af640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e88142520 con 0x7f9e881430c0 2024-03-25T18:20:53.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.731+0000 7f9e8c9af640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e880b28a0 con 0x7f9e8813f710 2024-03-25T18:20:53.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.731+0000 7f9e8c9af640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e880b2a30 con 0x7f9e880a98f0 2024-03-25T18:20:53.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.731+0000 7f9e8c9af640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:53.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.731+0000 7f9e8c9af640 10 monclient(hunting): authenticate will time out at 7947.917405s 2024-03-25T18:20:53.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.736+0000 7f9e8659c640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f9e880a98f0 0x7f9e880a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:34548/0 (socket says 172.21.15.63:34548) 2024-03-25T18:20:53.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.736+0000 7f9e8659c640 1 -- 172.21.15.63:0/277360940 learned_addr learned my addr 172.21.15.63:0/277360940 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:20:53.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.745+0000 7f9e85d9b640 1 -- 172.21.15.63:0/277360940 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174124091 0 0) 0x7f9e880b2a30 con 0x7f9e880a98f0 2024-03-25T18:20:53.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.745+0000 7f9e85d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:53.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.745+0000 7f9e85d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:53.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.745+0000 7f9e85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:53.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.745+0000 7f9e85d9b640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:20:53.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.745+0000 7f9e85d9b640 1 -- 172.21.15.63:0/277360940 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e74003520 con 0x7f9e880a98f0 2024-03-25T18:20:53.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.761+0000 7f9e85d9b640 1 -- 172.21.15.63:0/277360940 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710328481 0 0) 0x7f9e880b28a0 con 0x7f9e8813f710 2024-03-25T18:20:53.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.761+0000 7f9e85d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:53.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.761+0000 7f9e85d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:53.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.761+0000 7f9e85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:53.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.761+0000 7f9e85d9b640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:20:53.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.761+0000 7f9e85d9b640 1 -- 172.21.15.63:0/277360940 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e880b2a30 con 0x7f9e8813f710 2024-03-25T18:20:53.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.763+0000 7f9e85d9b640 1 -- 172.21.15.63:0/277360940 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201759474 0 0) 0x7f9e88142520 con 0x7f9e881430c0 2024-03-25T18:20:53.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.763+0000 7f9e85d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:53.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.763+0000 7f9e85d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:53.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.763+0000 7f9e85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:53.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.763+0000 7f9e85d9b640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:20:53.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.763+0000 7f9e85d9b640 1 -- 172.21.15.63:0/277360940 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e880b28a0 con 0x7f9e881430c0 2024-03-25T18:20:53.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e85d9b640 1 -- 172.21.15.63:0/277360940 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2392535276 0 0) 0x7f9e74003520 con 0x7f9e880a98f0 2024-03-25T18:20:53.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e85d9b640 1 -- 172.21.15.63:0/277360940 >> v1:172.21.15.81:6789/0 conn(0x7f9e8813f710 legacy=0x7f9e881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:53.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e85d9b640 1 -- 172.21.15.63:0/277360940 >> v1:172.21.15.63:6789/0 conn(0x7f9e881430c0 legacy=0x7f9e8814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:53.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e85d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:20:53.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e85d9b640 1 monclient: found mon.noname-b 2024-03-25T18:20:53.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e85d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:53.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e85d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:20:53.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e85d9b640 1 -- 172.21.15.63:0/277360940 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e880b44c0 con 0x7f9e880a98f0 2024-03-25T18:20:53.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e85d9b640 10 monclient: _finish_auth 0 2024-03-25T18:20:53.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e85d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:20:53.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e85d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:53.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e85d9b640 1 -- 172.21.15.63:0/277360940 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9e78001690 con 0x7f9e880a98f0 2024-03-25T18:20:53.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e85d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:53.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e85d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:53.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e85d9b640 10 monclient: dump: 2024-03-25T18:20:53.769 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:53.769 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:53.769 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:53.769 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:53.769 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:53.769 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:53.769 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:53.770 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:53.770 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:53.770 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:53.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e8c9af640 5 monclient: authenticate success, global_id 4583 2024-03-25T18:20:53.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.766+0000 7f9e8c9af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:53.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.777+0000 7f9e85d9b640 1 -- 172.21.15.63:0/277360940 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e780017f0 con 0x7f9e880a98f0 2024-03-25T18:20:53.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.777+0000 7f9e85d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:53.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.777+0000 7f9e85d9b640 1 -- 172.21.15.63:0/277360940 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9e78001c00 con 0x7f9e880a98f0 2024-03-25T18:20:53.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.777+0000 7f9e8c9af640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:53.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.777+0000 7f9e8c9af640 4 set_mon_vals no callback set 2024-03-25T18:20:53.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.777+0000 7f9e85d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:53.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.777+0000 7f9e85d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:53.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.777+0000 7f9e85d9b640 10 monclient: dump: 2024-03-25T18:20:53.779 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:53.779 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:53.779 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:53.780 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:53.780 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:53.780 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:53.780 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:53.780 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:53.780 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:53.780 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:53.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.778+0000 7f9e8c9af640 10 monclient: shutdown 2024-03-25T18:20:53.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.778+0000 7f9e8c9af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:53.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.778+0000 7f9e8c9af640 1 -- 172.21.15.63:0/277360940 >> v1:172.21.15.63:6790/0 conn(0x7f9e880a98f0 legacy=0x7f9e880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:53.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.778+0000 7f9e8c9af640 1 -- 172.21.15.63:0/277360940 shutdown_connections 2024-03-25T18:20:53.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.778+0000 7f9e8c9af640 1 -- 172.21.15.63:0/277360940 >> 172.21.15.63:0/277360940 conn(0x7f9e880b7ea0 msgr2=0x7f9e880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:53.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.778+0000 7f9e8c9af640 1 -- 172.21.15.63:0/277360940 shutdown_connections 2024-03-25T18:20:53.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.778+0000 7f9e8c9af640 1 -- 172.21.15.63:0/277360940 wait complete. 2024-03-25T18:20:53.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.779+0000 7f9e8c9af640 10 monclient: build_initial_monmap 2024-03-25T18:20:53.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.779+0000 7f9e8c9af640 10 monclient: monmap: 2024-03-25T18:20:53.782 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:20:53.782 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:53.783 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:20:53.783 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:20:53.783 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:53.783 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:20:53.783 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:20:53.783 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:20:53.783 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:20:53.783 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:53.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.780+0000 7f9e8c9af640 1 librados: starting msgr at 2024-03-25T18:20:53.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.780+0000 7f9e8c9af640 1 librados: starting objecter 2024-03-25T18:20:53.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.780+0000 7f9e8c9af640 1 Processor -- start 2024-03-25T18:20:53.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.780+0000 7f9e8c9af640 1 -- start start 2024-03-25T18:20:53.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.780+0000 7f9e8c9af640 1 librados: setting wanted keys 2024-03-25T18:20:53.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.780+0000 7f9e8c9af640 1 librados: calling monclient init 2024-03-25T18:20:53.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.780+0000 7f9e8c9af640 10 monclient: init 2024-03-25T18:20:53.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.781+0000 7f9e8c9af640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:53.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.781+0000 7f9e8c9af640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:53.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.781+0000 7f9e8c9af640 10 monclient(hunting): picked mon.noname-a con 0x7f9e880a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:20:53.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.781+0000 7f9e8c9af640 10 monclient(hunting): picked mon.noname-b con 0x7f9e8813f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:20:53.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.781+0000 7f9e8c9af640 10 monclient(hunting): picked mon.noname-c con 0x7f9e881430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:20:53.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.781+0000 7f9e8c9af640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e88070490 con 0x7f9e880a98f0 2024-03-25T18:20:53.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.781+0000 7f9e8c9af640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e8806e0a0 con 0x7f9e8813f710 2024-03-25T18:20:53.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.781+0000 7f9e8c9af640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e8806e230 con 0x7f9e881430c0 2024-03-25T18:20:53.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.781+0000 7f9e8c9af640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:53.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.781+0000 7f9e8c9af640 10 monclient(hunting): authenticate will time out at 7947.968098s 2024-03-25T18:20:53.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.799+0000 7f9e86d9d640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f9e881430c0 0x7f9e881e64d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:34628/0 (socket says 172.21.15.63:34628) 2024-03-25T18:20:53.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.799+0000 7f9e86d9d640 1 -- 172.21.15.63:0/4127048332 learned_addr learned my addr 172.21.15.63:0/4127048332 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:20:53.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.801+0000 7f9e7f7fe640 10 client.?.objecter ms_handle_connect 0x7f9e881430c0 2024-03-25T18:20:53.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.801+0000 7f9e7f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:53.805 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.804+0000 7f9e7f7fe640 10 client.?.objecter ms_handle_connect 0x7f9e880a98f0 2024-03-25T18:20:53.805 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.804+0000 7f9e7f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:53.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.806+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318334328 0 0) 0x7f9e8806e230 con 0x7f9e881430c0 2024-03-25T18:20:53.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.806+0000 7f9e7f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:53.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.806+0000 7f9e7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:53.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.806+0000 7f9e7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:53.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.806+0000 7f9e7f7fe640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:20:53.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.806+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e68004eb0 con 0x7f9e881430c0 2024-03-25T18:20:53.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.806+0000 7f9e7f7fe640 10 client.?.objecter ms_handle_connect 0x7f9e8813f710 2024-03-25T18:20:53.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.806+0000 7f9e7f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:53.818 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.816+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1202246776 0 0) 0x7f9e68004eb0 con 0x7f9e881430c0 2024-03-25T18:20:53.818 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.816+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e8806e230 con 0x7f9e881430c0 2024-03-25T18:20:53.818 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.816+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9e80001c70 con 0x7f9e881430c0 2024-03-25T18:20:53.818 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.816+0000 7f9e7f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:53.818 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.816+0000 7f9e7f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:53.818 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.816+0000 7f9e7f7fe640 10 monclient(hunting): dump: 2024-03-25T18:20:53.818 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:53.818 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:53.818 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:53.819 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:53.819 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:53.819 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:53.819 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:53.819 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:53.819 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:53.819 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:53.819 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.816+0000 7f9e7f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:53.819 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.816+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514726128 0 0) 0x7f9e8806e0a0 con 0x7f9e8813f710 2024-03-25T18:20:53.819 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.816+0000 7f9e7f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:53.819 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.816+0000 7f9e7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:53.819 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.816+0000 7f9e7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:53.819 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.816+0000 7f9e7f7fe640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:20:53.819 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.816+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e68004eb0 con 0x7f9e8813f710 2024-03-25T18:20:53.820 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.817+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103485031 0 0) 0x7f9e88070490 con 0x7f9e880a98f0 2024-03-25T18:20:53.820 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.817+0000 7f9e7f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:53.820 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.817+0000 7f9e7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:53.820 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.817+0000 7f9e7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:53.820 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.817+0000 7f9e7f7fe640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:20:53.820 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.817+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e8806e0a0 con 0x7f9e880a98f0 2024-03-25T18:20:53.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3578785568 0 0) 0x7f9e8806e230 con 0x7f9e881430c0 2024-03-25T18:20:53.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 >> v1:172.21.15.81:6789/0 conn(0x7f9e8813f710 legacy=0x7f9e8806ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:53.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 >> v1:172.21.15.63:6789/0 conn(0x7f9e880a98f0 legacy=0x7f9e881df440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:53.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e7f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:53.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e7f7fe640 1 monclient: found mon.c 2024-03-25T18:20:53.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:53.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e7f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:20:53.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e88070b20 con 0x7f9e881430c0 2024-03-25T18:20:53.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e7f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:53.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e7f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:53.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:53.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e8c9af640 5 monclient: authenticate success, global_id 4586 2024-03-25T18:20:53.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e8c9af640 10 monclient: _renew_subs 2024-03-25T18:20:53.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e8c9af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:20:53.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e8c9af640 1 -- 172.21.15.63:0/4127048332 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e881dfb50 con 0x7f9e881430c0 2024-03-25T18:20:53.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e8c9af640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:53.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e8c9af640 10 monclient: _renew_subs 2024-03-25T18:20:53.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e8c9af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:20:53.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e8c9af640 1 -- 172.21.15.63:0/4127048332 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8806f580 con 0x7f9e881430c0 2024-03-25T18:20:53.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.827+0000 7f9e8c9af640 1 librados: init done 2024-03-25T18:20:53.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.828+0000 7f9e8c9af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:53.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.828+0000 7f9e8c9af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:53.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.828+0000 7f9e8c9af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:20:53.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:53.828+0000 7f9e8c9af640 1 -- 172.21.15.63:0/4127048332 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e88070d70 con 0x7f9e881430c0 2024-03-25T18:20:54.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.151+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e80003970 con 0x7f9e881430c0 2024-03-25T18:20:54.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.151+0000 7f9e7f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:54.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.151+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9e80003dc0 con 0x7f9e881430c0 2024-03-25T18:20:54.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.151+0000 7f9e7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:54.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.151+0000 7f9e7f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:54.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.151+0000 7f9e7f7fe640 10 monclient: dump: 2024-03-25T18:20:54.152 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:20:54.152 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:20:54.152 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:54.152 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:20:54.153 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:54.153 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:20:54.153 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:20:54.153 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:20:54.153 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:20:54.153 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:20:54.153 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.151+0000 7f9e8559a640 4 set_mon_vals no callback set 2024-03-25T18:20:54.153 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.152+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f9e80003fe0 con 0x7f9e881430c0 2024-03-25T18:20:54.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.152+0000 7f9e8659c640 10 monclient: get_auth_request con 0x7f9e6806de80 auth_method 0 2024-03-25T18:20:54.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.153+0000 7f9e7f7fe640 10 client.4586.objecter ms_handle_connect 0x7f9e6806de80 2024-03-25T18:20:54.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.159+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f9e80054b00 con 0x7f9e881430c0 2024-03-25T18:20:54.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.159+0000 7f9e7f7fe640 10 client.4586.objecter ms_dispatch 0x7f9e88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:54.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.159+0000 7f9e7f7fe640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:54.160 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.159+0000 7f9e7f7fe640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:54.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.159+0000 7f9e7f7fe640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-25T18:20:54.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.159+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9e80050610 con 0x7f9e881430c0 2024-03-25T18:20:54.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.159+0000 7f9e7f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:54.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.159+0000 7f9e7f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:54.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.412+0000 7f9e8c9af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:54.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.412+0000 7f9e8c9af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:54.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.412+0000 7f9e8c9af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:20:54.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.412+0000 7f9e8c9af640 1 -- 172.21.15.63:0/4127048332 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e880627b0 con 0x7f9e881430c0 2024-03-25T18:20:54.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.439+0000 7f9e7f7fe640 1 -- 172.21.15.63:0/4127048332 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f9e80052540 con 0x7f9e881430c0 2024-03-25T18:20:54.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.439+0000 7f9e7f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:54.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.439+0000 7f9e7f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:54.440 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:20:54.440 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:54.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.441+0000 7f9e8c9af640 10 librados: watch_flush enter 2024-03-25T18:20:54.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.441+0000 7f9e8c9af640 10 librados: watch_flush exit 2024-03-25T18:20:54.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.441+0000 7f9e8c9af640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-25T18:20:54.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.441+0000 7f9e8c9af640 10 client.4586.objecter successfully canceled tick 2024-03-25T18:20:54.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.441+0000 7f9e8c9af640 1 -- 172.21.15.63:0/4127048332 >> v1:172.21.15.81:6800/3640340873 conn(0x7f9e6806de80 legacy=0x7f9e68070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:54.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.441+0000 7f9e8c9af640 10 monclient: shutdown 2024-03-25T18:20:54.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.441+0000 7f9e8c9af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:54.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.441+0000 7f9e8c9af640 1 -- 172.21.15.63:0/4127048332 >> v1:172.21.15.63:6790/0 conn(0x7f9e881430c0 legacy=0x7f9e881e64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:54.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.441+0000 7f9e8c9af640 1 -- 172.21.15.63:0/4127048332 shutdown_connections 2024-03-25T18:20:54.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.441+0000 7f9e8c9af640 1 -- 172.21.15.63:0/4127048332 >> 172.21.15.63:0/4127048332 conn(0x7f9e880b7ea0 msgr2=0x7f9e88074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:54.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.441+0000 7f9e8c9af640 1 -- 172.21.15.63:0/4127048332 shutdown_connections 2024-03-25T18:20:54.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.441+0000 7f9e8c9af640 1 -- 172.21.15.63:0/4127048332 wait complete. 2024-03-25T18:20:54.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:20:54.441+0000 7f9e8c9af640 1 librados: shutdown 2024-03-25T18:20:54.454 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:01.259 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:01.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.375+0000 7f2123cfb640 10 monclient: get_monmap_and_config 2024-03-25T18:21:01.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.375+0000 7f2123cfb640 10 monclient: build_initial_monmap 2024-03-25T18:21:01.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.375+0000 7f2123cfb640 10 monclient: monmap: 2024-03-25T18:21:01.376 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:21:01.376 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:01.376 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:21:01.376370+0000 2024-03-25T18:21:01.376 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:21:01.376370+0000 2024-03-25T18:21:01.376 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:01.376 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:01.376 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:21:01.377 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:21:01.377 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:21:01.377 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:01.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.376+0000 7f2123cfb640 1 Processor -- start 2024-03-25T18:21:01.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.376+0000 7f2123cfb640 1 -- start start 2024-03-25T18:21:01.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.376+0000 7f2123cfb640 10 monclient: init 2024-03-25T18:21:01.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.377+0000 7f2123cfb640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:01.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.377+0000 7f2123cfb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:01.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.377+0000 7f2123cfb640 10 monclient(hunting): picked mon.noname-c con 0x7f211c0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:21:01.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.377+0000 7f2123cfb640 10 monclient(hunting): picked mon.noname-a con 0x7f211c13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:21:01.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.377+0000 7f2123cfb640 10 monclient(hunting): picked mon.noname-b con 0x7f211c1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:21:01.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.377+0000 7f2123cfb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211c142520 con 0x7f211c13f710 2024-03-25T18:21:01.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.377+0000 7f2123cfb640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211c0b28a0 con 0x7f211c0a98f0 2024-03-25T18:21:01.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.377+0000 7f2123cfb640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211c0b2a30 con 0x7f211c1430c0 2024-03-25T18:21:01.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.377+0000 7f2123cfb640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:01.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.377+0000 7f2123cfb640 10 monclient(hunting): authenticate will time out at 7955.563456s 2024-03-25T18:21:01.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.390+0000 7f2121296640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f211c13f710 0x7f211c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:43008/0 (socket says 172.21.15.63:43008) 2024-03-25T18:21:01.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.390+0000 7f2121296640 1 -- 172.21.15.63:0/2553879006 learned_addr learned my addr 172.21.15.63:0/2553879006 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:21:01.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.407+0000 7f2120a95640 1 -- 172.21.15.63:0/2553879006 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108285895 0 0) 0x7f211c142520 con 0x7f211c13f710 2024-03-25T18:21:01.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.407+0000 7f2120a95640 10 monclient(hunting): _negotiate 2024-03-25T18:21:01.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.407+0000 7f2120a95640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:01.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.407+0000 7f2120a95640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:01.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.407+0000 7f2120a95640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:21:01.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.407+0000 7f2120a95640 1 -- 172.21.15.63:0/2553879006 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2108003520 con 0x7f211c13f710 2024-03-25T18:21:01.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2120a95640 1 -- 172.21.15.63:0/2553879006 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1983581881 0 0) 0x7f2108003520 con 0x7f211c13f710 2024-03-25T18:21:01.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2120a95640 1 -- 172.21.15.63:0/2553879006 >> v1:172.21.15.63:6790/0 conn(0x7f211c1430c0 legacy=0x7f211c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:01.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2120a95640 1 -- 172.21.15.63:0/2553879006 >> v1:172.21.15.81:6789/0 conn(0x7f211c0a98f0 legacy=0x7f211c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:01.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2120a95640 10 monclient: _finish_hunting 0 2024-03-25T18:21:01.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2120a95640 1 monclient: found mon.noname-a 2024-03-25T18:21:01.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2120a95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:01.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2120a95640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:21:01.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2120a95640 1 -- 172.21.15.63:0/2553879006 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c0b44c0 con 0x7f211c13f710 2024-03-25T18:21:01.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2120a95640 10 monclient: _finish_auth 0 2024-03-25T18:21:01.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2120a95640 10 monclient: _check_auth_tickets 2024-03-25T18:21:01.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2120a95640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:01.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2120a95640 1 -- 172.21.15.63:0/2553879006 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f210c001690 con 0x7f211c13f710 2024-03-25T18:21:01.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2120a95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:01.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2120a95640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:01.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2120a95640 10 monclient: dump: 2024-03-25T18:21:01.473 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:01.473 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:01.473 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:01.473 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:01.473 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:01.473 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:01.473 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:01.473 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:01.473 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:01.474 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:01.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2123cfb640 5 monclient: authenticate success, global_id 4581 2024-03-25T18:21:01.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.471+0000 7f2123cfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:01.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.482+0000 7f2120a95640 1 -- 172.21.15.63:0/2553879006 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f210c0017f0 con 0x7f211c13f710 2024-03-25T18:21:01.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.482+0000 7f2120a95640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:01.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.482+0000 7f2120a95640 1 -- 172.21.15.63:0/2553879006 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f210c001c00 con 0x7f211c13f710 2024-03-25T18:21:01.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.482+0000 7f2123cfb640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:01.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.482+0000 7f2123cfb640 4 set_mon_vals no callback set 2024-03-25T18:21:01.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.482+0000 7f2120a95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:01.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.482+0000 7f2120a95640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:01.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.482+0000 7f2120a95640 10 monclient: dump: 2024-03-25T18:21:01.484 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:01.484 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:01.484 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:01.484 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:01.484 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:01.484 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:01.484 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:01.484 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:01.484 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:01.484 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:01.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.483+0000 7f2123cfb640 10 monclient: shutdown 2024-03-25T18:21:01.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.483+0000 7f2123cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:01.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.483+0000 7f2123cfb640 1 -- 172.21.15.63:0/2553879006 >> v1:172.21.15.63:6789/0 conn(0x7f211c13f710 legacy=0x7f211c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:01.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.483+0000 7f2123cfb640 1 -- 172.21.15.63:0/2553879006 shutdown_connections 2024-03-25T18:21:01.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.483+0000 7f2123cfb640 1 -- 172.21.15.63:0/2553879006 >> 172.21.15.63:0/2553879006 conn(0x7f211c0b7ea0 msgr2=0x7f211c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:01.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.483+0000 7f2123cfb640 1 -- 172.21.15.63:0/2553879006 shutdown_connections 2024-03-25T18:21:01.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.483+0000 7f2123cfb640 1 -- 172.21.15.63:0/2553879006 wait complete. 2024-03-25T18:21:01.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.484+0000 7f2123cfb640 10 monclient: build_initial_monmap 2024-03-25T18:21:01.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.484+0000 7f2123cfb640 10 monclient: monmap: 2024-03-25T18:21:01.486 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:21:01.486 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:01.486 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:21:01.486 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:21:01.486 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:01.486 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:21:01.486 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:21:01.486 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:21:01.486 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:21:01.486 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:01.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.485+0000 7f2123cfb640 1 librados: starting msgr at 2024-03-25T18:21:01.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.485+0000 7f2123cfb640 1 librados: starting objecter 2024-03-25T18:21:01.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.485+0000 7f2123cfb640 1 Processor -- start 2024-03-25T18:21:01.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.485+0000 7f2123cfb640 1 -- start start 2024-03-25T18:21:01.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.485+0000 7f2123cfb640 1 librados: setting wanted keys 2024-03-25T18:21:01.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.485+0000 7f2123cfb640 1 librados: calling monclient init 2024-03-25T18:21:01.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.485+0000 7f2123cfb640 10 monclient: init 2024-03-25T18:21:01.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.485+0000 7f2123cfb640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:01.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.485+0000 7f2123cfb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:01.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.485+0000 7f2123cfb640 10 monclient(hunting): picked mon.noname-b con 0x7f211c0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:21:01.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.485+0000 7f2123cfb640 10 monclient(hunting): picked mon.noname-a con 0x7f211c13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:21:01.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.485+0000 7f2123cfb640 10 monclient(hunting): picked mon.noname-c con 0x7f211c1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:21:01.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.485+0000 7f2123cfb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211c0abd70 con 0x7f211c13f710 2024-03-25T18:21:01.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.485+0000 7f2123cfb640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211c1d7710 con 0x7f211c0a98f0 2024-03-25T18:21:01.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.486+0000 7f2123cfb640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211c1d8340 con 0x7f211c1430c0 2024-03-25T18:21:01.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.486+0000 7f2123cfb640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:01.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.486+0000 7f2123cfb640 10 monclient(hunting): authenticate will time out at 7955.672304s 2024-03-25T18:21:01.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.502+0000 7f2121296640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f211c13f710 0x7f211c1d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:43042/0 (socket says 172.21.15.63:43042) 2024-03-25T18:21:01.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.502+0000 7f2121296640 1 -- 172.21.15.63:0/1903795669 learned_addr learned my addr 172.21.15.63:0/1903795669 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:21:01.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.506+0000 7f21127fc640 10 client.?.objecter ms_handle_connect 0x7f211c13f710 2024-03-25T18:21:01.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.506+0000 7f21127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:01.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.515+0000 7f21127fc640 1 -- 172.21.15.63:0/1903795669 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834724939 0 0) 0x7f211c0abd70 con 0x7f211c13f710 2024-03-25T18:21:01.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.515+0000 7f21127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:01.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.515+0000 7f21127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:01.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.515+0000 7f21127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:01.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.515+0000 7f21127fc640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:21:01.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.515+0000 7f21127fc640 1 -- 172.21.15.63:0/1903795669 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2100004eb0 con 0x7f211c13f710 2024-03-25T18:21:01.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.529+0000 7f21127fc640 1 -- 172.21.15.63:0/1903795669 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 98543870 0 0) 0x7f2100004eb0 con 0x7f211c13f710 2024-03-25T18:21:01.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.529+0000 7f21127fc640 1 -- 172.21.15.63:0/1903795669 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f211c0abd70 con 0x7f211c13f710 2024-03-25T18:21:01.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.529+0000 7f21127fc640 1 -- 172.21.15.63:0/1903795669 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f210c001c00 con 0x7f211c13f710 2024-03-25T18:21:01.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.529+0000 7f21127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:01.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.529+0000 7f21127fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:01.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.529+0000 7f21127fc640 10 monclient(hunting): dump: 2024-03-25T18:21:01.531 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:01.531 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:01.531 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:01.531 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:01.531 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:01.531 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:01.532 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:01.532 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:01.532 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:01.532 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:01.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.529+0000 7f21127fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:01.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.534+0000 7f21127fc640 10 client.?.objecter ms_handle_connect 0x7f211c1430c0 2024-03-25T18:21:01.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.534+0000 7f21127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:01.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.537+0000 7f21127fc640 10 client.?.objecter ms_handle_connect 0x7f211c0a98f0 2024-03-25T18:21:01.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.537+0000 7f21127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:01.546 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f21127fc640 1 -- 172.21.15.63:0/1903795669 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 958069001 0 0) 0x7f211c0abd70 con 0x7f211c13f710 2024-03-25T18:21:01.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f21127fc640 1 -- 172.21.15.63:0/1903795669 >> v1:172.21.15.63:6790/0 conn(0x7f211c1430c0 legacy=0x7f211c1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:01.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f21127fc640 1 -- 172.21.15.63:0/1903795669 >> v1:172.21.15.81:6789/0 conn(0x7f211c0a98f0 legacy=0x7f211c137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:01.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f21127fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:01.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f21127fc640 1 monclient: found mon.a 2024-03-25T18:21:01.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f21127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:01.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f21127fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:21:01.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f21127fc640 1 -- 172.21.15.63:0/1903795669 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c1d84d0 con 0x7f211c13f710 2024-03-25T18:21:01.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f21127fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:01.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f21127fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:01.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f21127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:01.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f2123cfb640 5 monclient: authenticate success, global_id 4584 2024-03-25T18:21:01.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f2123cfb640 10 monclient: _renew_subs 2024-03-25T18:21:01.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f2123cfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:21:01.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f2123cfb640 1 -- 172.21.15.63:0/1903795669 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f211c1d41e0 con 0x7f211c13f710 2024-03-25T18:21:01.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f2123cfb640 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:01.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f2123cfb640 10 monclient: _renew_subs 2024-03-25T18:21:01.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f2123cfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:21:01.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f2123cfb640 1 -- 172.21.15.63:0/1903795669 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f211c1d5150 con 0x7f211c13f710 2024-03-25T18:21:01.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.545+0000 7f2123cfb640 1 librados: init done 2024-03-25T18:21:01.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.546+0000 7f2123cfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:01.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.546+0000 7f2123cfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:01.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.546+0000 7f2123cfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:21:01.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.546+0000 7f2123cfb640 1 -- 172.21.15.63:0/1903795669 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f211c1d7900 con 0x7f211c13f710 2024-03-25T18:21:01.860 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.859+0000 7f21127fc640 1 -- 172.21.15.63:0/1903795669 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f210c004a90 con 0x7f211c13f710 2024-03-25T18:21:01.860 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.859+0000 7f21127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:01.860 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.859+0000 7f21127fc640 1 -- 172.21.15.63:0/1903795669 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f210c004ee0 con 0x7f211c13f710 2024-03-25T18:21:01.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.859+0000 7f21127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:01.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.859+0000 7f21127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:01.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.859+0000 7f21127fc640 10 monclient: dump: 2024-03-25T18:21:01.861 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:01.861 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:01.861 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:01.861 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:01.861 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:01.861 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:01.861 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:01.861 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:01.861 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:01.861 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:01.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.859+0000 7f2113fff640 4 set_mon_vals no callback set 2024-03-25T18:21:01.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.860+0000 7f21127fc640 1 -- 172.21.15.63:0/1903795669 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f210c005100 con 0x7f211c13f710 2024-03-25T18:21:01.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.861+0000 7f2121a97640 10 monclient: get_auth_request con 0x7f210006dc30 auth_method 0 2024-03-25T18:21:01.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.861+0000 7f21127fc640 10 client.4584.objecter ms_handle_connect 0x7f210006dc30 2024-03-25T18:21:01.869 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.868+0000 7f21127fc640 1 -- 172.21.15.63:0/1903795669 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f210c054a20 con 0x7f211c13f710 2024-03-25T18:21:01.869 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.868+0000 7f21127fc640 10 client.4584.objecter ms_dispatch 0x7f211c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:01.869 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.868+0000 7f21127fc640 3 client.4584.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:01.869 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.868+0000 7f21127fc640 3 client.4584.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:01.869 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.868+0000 7f21127fc640 20 client.4584.objecter dump_active .. 0 homeless 2024-03-25T18:21:01.870 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.868+0000 7f21127fc640 1 -- 172.21.15.63:0/1903795669 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f210c050530 con 0x7f211c13f710 2024-03-25T18:21:01.870 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.868+0000 7f21127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:01.870 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:01.868+0000 7f21127fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:02.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.113+0000 7f2123cfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:02.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.113+0000 7f2123cfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:02.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.113+0000 7f2123cfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:21:02.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.113+0000 7f2123cfb640 1 -- 172.21.15.63:0/1903795669 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f211c0627b0 con 0x7f211c13f710 2024-03-25T18:21:02.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.147+0000 7f21127fc640 1 -- 172.21.15.63:0/1903795669 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f210c052460 con 0x7f211c13f710 2024-03-25T18:21:02.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.147+0000 7f21127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:02.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.147+0000 7f21127fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:02.149 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:21:02.149 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:02.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.150+0000 7f2123cfb640 10 librados: watch_flush enter 2024-03-25T18:21:02.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.150+0000 7f2123cfb640 10 librados: watch_flush exit 2024-03-25T18:21:02.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.150+0000 7f2123cfb640 20 client.4584.objecter shutdown clearing up homeless session... 2024-03-25T18:21:02.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.150+0000 7f2123cfb640 10 client.4584.objecter successfully canceled tick 2024-03-25T18:21:02.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.150+0000 7f2123cfb640 1 -- 172.21.15.63:0/1903795669 >> v1:172.21.15.81:6800/3640340873 conn(0x7f210006dc30 legacy=0x7f21000700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:02.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.150+0000 7f2123cfb640 10 monclient: shutdown 2024-03-25T18:21:02.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.150+0000 7f2123cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:02.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.150+0000 7f2123cfb640 1 -- 172.21.15.63:0/1903795669 >> v1:172.21.15.63:6789/0 conn(0x7f211c13f710 legacy=0x7f211c1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:02.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.150+0000 7f2123cfb640 1 -- 172.21.15.63:0/1903795669 shutdown_connections 2024-03-25T18:21:02.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.150+0000 7f2123cfb640 1 -- 172.21.15.63:0/1903795669 >> 172.21.15.63:0/1903795669 conn(0x7f211c0b7ea0 msgr2=0x7f211c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:02.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.150+0000 7f2123cfb640 1 -- 172.21.15.63:0/1903795669 shutdown_connections 2024-03-25T18:21:02.153 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.150+0000 7f2123cfb640 1 -- 172.21.15.63:0/1903795669 wait complete. 2024-03-25T18:21:02.153 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:02.150+0000 7f2123cfb640 1 librados: shutdown 2024-03-25T18:21:02.163 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:04.756 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:46.448 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:04.766 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:46.458 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:04.804 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:46.496 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:04.807 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:08:46.468 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:04.811 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:08:46.472 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:04.837 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:46.529 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:04.844 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:08:46.504 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:04.848 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:46.540 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:04.876 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:46.569 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:08.969 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:09.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.087+0000 7fc5a038f640 10 monclient: get_monmap_and_config 2024-03-25T18:21:09.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.087+0000 7fc5a038f640 10 monclient: build_initial_monmap 2024-03-25T18:21:09.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.087+0000 7fc5a038f640 10 monclient: monmap: 2024-03-25T18:21:09.088 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:21:09.088 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:09.089 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:21:09.088438+0000 2024-03-25T18:21:09.089 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:21:09.088438+0000 2024-03-25T18:21:09.089 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:09.089 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:09.089 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:21:09.089 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:21:09.089 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:21:09.089 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:09.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.088+0000 7fc5a038f640 1 Processor -- start 2024-03-25T18:21:09.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.088+0000 7fc5a038f640 1 -- start start 2024-03-25T18:21:09.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.088+0000 7fc5a038f640 10 monclient: init 2024-03-25T18:21:09.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.089+0000 7fc5a038f640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:09.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.089+0000 7fc5a038f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:09.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.089+0000 7fc5a038f640 10 monclient(hunting): picked mon.noname-b con 0x7fc5980a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:21:09.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.089+0000 7fc5a038f640 10 monclient(hunting): picked mon.noname-a con 0x7fc59813f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:21:09.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.089+0000 7fc5a038f640 10 monclient(hunting): picked mon.noname-c con 0x7fc5981430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:21:09.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.089+0000 7fc5a038f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc598142520 con 0x7fc59813f710 2024-03-25T18:21:09.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.089+0000 7fc5a038f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5980b28a0 con 0x7fc5981430c0 2024-03-25T18:21:09.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.089+0000 7fc5a038f640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5980b2a30 con 0x7fc5980a98f0 2024-03-25T18:21:09.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.089+0000 7fc5a038f640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:09.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.089+0000 7fc5a038f640 10 monclient(hunting): authenticate will time out at 7963.275451s 2024-03-25T18:21:09.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.095+0000 7fc59e92c640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc5981430c0 0x7fc59814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:47686/0 (socket says 172.21.15.63:47686) 2024-03-25T18:21:09.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.095+0000 7fc59e92c640 1 -- 172.21.15.63:0/1605436124 learned_addr learned my addr 172.21.15.63:0/1605436124 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:21:09.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.108+0000 7fc59d129640 1 -- 172.21.15.63:0/1605436124 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760704836 0 0) 0x7fc5980b2a30 con 0x7fc5980a98f0 2024-03-25T18:21:09.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.108+0000 7fc59d129640 10 monclient(hunting): _negotiate 2024-03-25T18:21:09.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.108+0000 7fc59d129640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:09.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.108+0000 7fc59d129640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:09.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.108+0000 7fc59d129640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:21:09.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.108+0000 7fc59d129640 1 -- 172.21.15.63:0/1605436124 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc584003520 con 0x7fc5980a98f0 2024-03-25T18:21:09.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.110+0000 7fc59d129640 1 -- 172.21.15.63:0/1605436124 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757108667 0 0) 0x7fc5980b28a0 con 0x7fc5981430c0 2024-03-25T18:21:09.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.110+0000 7fc59d129640 10 monclient(hunting): _negotiate 2024-03-25T18:21:09.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.110+0000 7fc59d129640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:09.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.110+0000 7fc59d129640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:09.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.110+0000 7fc59d129640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:21:09.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.110+0000 7fc59d129640 1 -- 172.21.15.63:0/1605436124 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5980b2a30 con 0x7fc5981430c0 2024-03-25T18:21:09.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.110+0000 7fc59d129640 1 -- 172.21.15.63:0/1605436124 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043135520 0 0) 0x7fc598142520 con 0x7fc59813f710 2024-03-25T18:21:09.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.110+0000 7fc59d129640 10 monclient(hunting): _negotiate 2024-03-25T18:21:09.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.110+0000 7fc59d129640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:09.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.110+0000 7fc59d129640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:09.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.110+0000 7fc59d129640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:21:09.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.110+0000 7fc59d129640 1 -- 172.21.15.63:0/1605436124 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5980b28a0 con 0x7fc59813f710 2024-03-25T18:21:09.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc59d129640 1 -- 172.21.15.63:0/1605436124 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2568480829 0 0) 0x7fc584003520 con 0x7fc5980a98f0 2024-03-25T18:21:09.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc59d129640 1 -- 172.21.15.63:0/1605436124 >> v1:172.21.15.81:6789/0 conn(0x7fc5981430c0 legacy=0x7fc59814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:09.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc59d129640 1 -- 172.21.15.63:0/1605436124 >> v1:172.21.15.63:6789/0 conn(0x7fc59813f710 legacy=0x7fc5981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:09.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc59d129640 10 monclient: _finish_hunting 0 2024-03-25T18:21:09.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc59d129640 1 monclient: found mon.noname-b 2024-03-25T18:21:09.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc59d129640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:09.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc59d129640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:21:09.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc59d129640 1 -- 172.21.15.63:0/1605436124 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5980b44c0 con 0x7fc5980a98f0 2024-03-25T18:21:09.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc59d129640 10 monclient: _finish_auth 0 2024-03-25T18:21:09.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc59d129640 10 monclient: _check_auth_tickets 2024-03-25T18:21:09.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc59d129640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:09.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc59d129640 1 -- 172.21.15.63:0/1605436124 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc594001690 con 0x7fc5980a98f0 2024-03-25T18:21:09.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc59d129640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:09.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc59d129640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:09.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc59d129640 10 monclient: dump: 2024-03-25T18:21:09.118 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:09.118 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:09.118 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:09.118 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:09.118 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:09.118 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:09.118 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:09.118 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:09.118 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:09.118 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:09.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc5a038f640 5 monclient: authenticate success, global_id 4595 2024-03-25T18:21:09.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.115+0000 7fc5a038f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:09.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.120+0000 7fc59d129640 1 -- 172.21.15.63:0/1605436124 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5940017f0 con 0x7fc5980a98f0 2024-03-25T18:21:09.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.120+0000 7fc59d129640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:09.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.120+0000 7fc59d129640 1 -- 172.21.15.63:0/1605436124 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc594001c00 con 0x7fc5980a98f0 2024-03-25T18:21:09.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.120+0000 7fc59d129640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:09.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.120+0000 7fc59d129640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:09.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.120+0000 7fc59d129640 10 monclient: dump: 2024-03-25T18:21:09.121 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:09.121 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:09.121 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:09.121 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:09.122 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:09.122 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:09.122 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:09.122 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:09.122 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:09.122 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:09.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.120+0000 7fc5a038f640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:09.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.120+0000 7fc5a038f640 4 set_mon_vals no callback set 2024-03-25T18:21:09.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.121+0000 7fc5a038f640 10 monclient: shutdown 2024-03-25T18:21:09.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.121+0000 7fc5a038f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:09.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.121+0000 7fc5a038f640 1 -- 172.21.15.63:0/1605436124 >> v1:172.21.15.63:6790/0 conn(0x7fc5980a98f0 legacy=0x7fc5980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:09.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.121+0000 7fc5a038f640 1 -- 172.21.15.63:0/1605436124 shutdown_connections 2024-03-25T18:21:09.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.121+0000 7fc5a038f640 1 -- 172.21.15.63:0/1605436124 >> 172.21.15.63:0/1605436124 conn(0x7fc5980b7ea0 msgr2=0x7fc5980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:09.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.121+0000 7fc5a038f640 1 -- 172.21.15.63:0/1605436124 shutdown_connections 2024-03-25T18:21:09.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.121+0000 7fc5a038f640 1 -- 172.21.15.63:0/1605436124 wait complete. 2024-03-25T18:21:09.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.122+0000 7fc5a038f640 10 monclient: build_initial_monmap 2024-03-25T18:21:09.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.122+0000 7fc5a038f640 10 monclient: monmap: 2024-03-25T18:21:09.124 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:21:09.124 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:09.125 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:21:09.125 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:21:09.125 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:09.125 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:21:09.125 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:21:09.125 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:21:09.125 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:21:09.125 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:09.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.122+0000 7fc5a038f640 1 librados: starting msgr at 2024-03-25T18:21:09.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.122+0000 7fc5a038f640 1 librados: starting objecter 2024-03-25T18:21:09.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.122+0000 7fc5a038f640 1 Processor -- start 2024-03-25T18:21:09.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.122+0000 7fc5a038f640 1 -- start start 2024-03-25T18:21:09.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.122+0000 7fc5a038f640 1 librados: setting wanted keys 2024-03-25T18:21:09.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.122+0000 7fc5a038f640 1 librados: calling monclient init 2024-03-25T18:21:09.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.122+0000 7fc5a038f640 10 monclient: init 2024-03-25T18:21:09.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.123+0000 7fc5a038f640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:09.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.123+0000 7fc5a038f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:09.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.123+0000 7fc5a038f640 10 monclient(hunting): picked mon.noname-c con 0x7fc5980a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:21:09.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.123+0000 7fc5a038f640 10 monclient(hunting): picked mon.noname-a con 0x7fc59813f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:21:09.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.123+0000 7fc5a038f640 10 monclient(hunting): picked mon.noname-b con 0x7fc5981430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:21:09.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.123+0000 7fc5a038f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc598135610 con 0x7fc59813f710 2024-03-25T18:21:09.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.123+0000 7fc5a038f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc598137230 con 0x7fc5981430c0 2024-03-25T18:21:09.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.123+0000 7fc5a038f640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc598135cf0 con 0x7fc5980a98f0 2024-03-25T18:21:09.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.123+0000 7fc5a038f640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:09.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.123+0000 7fc5a038f640 10 monclient(hunting): authenticate will time out at 7963.309395s 2024-03-25T18:21:09.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.135+0000 7fc59e92c640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc5981430c0 0x7fc5981e2ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:47712/0 (socket says 172.21.15.63:47712) 2024-03-25T18:21:09.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.135+0000 7fc59e92c640 1 -- 172.21.15.63:0/391941752 learned_addr learned my addr 172.21.15.63:0/391941752 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:21:09.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.136+0000 7fc58effd640 10 client.?.objecter ms_handle_connect 0x7fc5981430c0 2024-03-25T18:21:09.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.136+0000 7fc58effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:09.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.140+0000 7fc58effd640 10 client.?.objecter ms_handle_connect 0x7fc5980a98f0 2024-03-25T18:21:09.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.141+0000 7fc58effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:09.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.143+0000 7fc58effd640 10 client.?.objecter ms_handle_connect 0x7fc59813f710 2024-03-25T18:21:09.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.143+0000 7fc58effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:09.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.150+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361674820 0 0) 0x7fc598135cf0 con 0x7fc5980a98f0 2024-03-25T18:21:09.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.150+0000 7fc58effd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:09.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.150+0000 7fc58effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:09.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.150+0000 7fc58effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:09.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.150+0000 7fc58effd640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:21:09.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.150+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc57c004eb0 con 0x7fc5980a98f0 2024-03-25T18:21:09.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.154+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132241926 0 0) 0x7fc598137230 con 0x7fc5981430c0 2024-03-25T18:21:09.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.154+0000 7fc58effd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:09.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.154+0000 7fc58effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:09.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.154+0000 7fc58effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:09.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.154+0000 7fc58effd640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:21:09.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.154+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc598135cf0 con 0x7fc5981430c0 2024-03-25T18:21:09.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.155+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135598810 0 0) 0x7fc598135610 con 0x7fc59813f710 2024-03-25T18:21:09.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.155+0000 7fc58effd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:09.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.155+0000 7fc58effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:09.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.155+0000 7fc58effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:09.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.155+0000 7fc58effd640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:21:09.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.156+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc598137230 con 0x7fc59813f710 2024-03-25T18:21:09.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.160+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1758070940 0 0) 0x7fc57c004eb0 con 0x7fc5980a98f0 2024-03-25T18:21:09.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.160+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc598135610 con 0x7fc5980a98f0 2024-03-25T18:21:09.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.160+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc594001b00 con 0x7fc5980a98f0 2024-03-25T18:21:09.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.160+0000 7fc58effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:09.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.160+0000 7fc58effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:09.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.160+0000 7fc58effd640 10 monclient(hunting): dump: 2024-03-25T18:21:09.162 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:09.162 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:09.162 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:09.162 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:09.162 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:09.162 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:09.163 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:09.163 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:09.163 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:09.163 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:09.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.160+0000 7fc58effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:09.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3530145684 0 0) 0x7fc598135610 con 0x7fc5980a98f0 2024-03-25T18:21:09.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 >> v1:172.21.15.81:6789/0 conn(0x7fc5981430c0 legacy=0x7fc5981e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:09.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 >> v1:172.21.15.63:6789/0 conn(0x7fc59813f710 legacy=0x7fc5981dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:09.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc58effd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:09.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc58effd640 1 monclient: found mon.c 2024-03-25T18:21:09.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc58effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:09.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc58effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:21:09.170 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc598135e80 con 0x7fc5980a98f0 2024-03-25T18:21:09.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc58effd640 10 monclient: _finish_auth 0 2024-03-25T18:21:09.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc58effd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:09.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc58effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:09.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc5a038f640 5 monclient: authenticate success, global_id 4598 2024-03-25T18:21:09.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc5a038f640 10 monclient: _renew_subs 2024-03-25T18:21:09.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc5a038f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:21:09.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc5a038f640 1 -- 172.21.15.63:0/391941752 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc598131660 con 0x7fc5980a98f0 2024-03-25T18:21:09.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc5a038f640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:09.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc5a038f640 10 monclient: _renew_subs 2024-03-25T18:21:09.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc5a038f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:21:09.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc5a038f640 1 -- 172.21.15.63:0/391941752 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5980b7960 con 0x7fc5980a98f0 2024-03-25T18:21:09.171 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.169+0000 7fc5a038f640 1 librados: init done 2024-03-25T18:21:09.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.170+0000 7fc5a038f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:09.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.170+0000 7fc5a038f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:09.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.170+0000 7fc5a038f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:21:09.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.170+0000 7fc5a038f640 1 -- 172.21.15.63:0/391941752 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5980627b0 con 0x7fc5980a98f0 2024-03-25T18:21:09.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.493+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5940038a0 con 0x7fc5980a98f0 2024-03-25T18:21:09.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.493+0000 7fc58effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:09.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.493+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc594003bb0 con 0x7fc5980a98f0 2024-03-25T18:21:09.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.493+0000 7fc58effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:09.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.493+0000 7fc58effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:09.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.493+0000 7fc58effd640 10 monclient: dump: 2024-03-25T18:21:09.495 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:09.495 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:09.495 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:09.495 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:09.496 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:09.496 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:09.496 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:09.496 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:09.496 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:09.496 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:09.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.493+0000 7fc59c928640 4 set_mon_vals no callback set 2024-03-25T18:21:09.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.494+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fc594019b80 con 0x7fc5980a98f0 2024-03-25T18:21:09.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.495+0000 7fc59d92a640 10 monclient: get_auth_request con 0x7fc57c06ddf0 auth_method 0 2024-03-25T18:21:09.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.495+0000 7fc58effd640 10 client.4598.objecter ms_handle_connect 0x7fc57c06ddf0 2024-03-25T18:21:09.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.501+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fc5940536a0 con 0x7fc5980a98f0 2024-03-25T18:21:09.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.501+0000 7fc58effd640 10 client.4598.objecter ms_dispatch 0x7fc598000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:09.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.501+0000 7fc58effd640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:09.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.501+0000 7fc58effd640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:09.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.501+0000 7fc58effd640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-25T18:21:09.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.501+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc59404efe0 con 0x7fc5980a98f0 2024-03-25T18:21:09.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.501+0000 7fc58effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:09.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.501+0000 7fc58effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:09.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.753+0000 7fc5a038f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:09.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.753+0000 7fc5a038f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:09.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.753+0000 7fc5a038f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:21:09.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.753+0000 7fc5a038f640 1 -- 172.21.15.63:0/391941752 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5981351c0 con 0x7fc5980a98f0 2024-03-25T18:21:09.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.791+0000 7fc58effd640 1 -- 172.21.15.63:0/391941752 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fc5940510e0 con 0x7fc5980a98f0 2024-03-25T18:21:09.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.791+0000 7fc58effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:09.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.791+0000 7fc58effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:09.793 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:21:09.793 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:09.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.794+0000 7fc5a038f640 10 librados: watch_flush enter 2024-03-25T18:21:09.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.794+0000 7fc5a038f640 10 librados: watch_flush exit 2024-03-25T18:21:09.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.794+0000 7fc5a038f640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-25T18:21:09.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.794+0000 7fc5a038f640 10 client.4598.objecter successfully canceled tick 2024-03-25T18:21:09.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.794+0000 7fc5a038f640 1 -- 172.21.15.63:0/391941752 >> v1:172.21.15.81:6800/3640340873 conn(0x7fc57c06ddf0 legacy=0x7fc57c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:09.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.794+0000 7fc5a038f640 10 monclient: shutdown 2024-03-25T18:21:09.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.794+0000 7fc5a038f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:09.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.794+0000 7fc5a038f640 1 -- 172.21.15.63:0/391941752 >> v1:172.21.15.63:6790/0 conn(0x7fc5980a98f0 legacy=0x7fc598130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:09.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.794+0000 7fc5a038f640 1 -- 172.21.15.63:0/391941752 shutdown_connections 2024-03-25T18:21:09.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.794+0000 7fc5a038f640 1 -- 172.21.15.63:0/391941752 >> 172.21.15.63:0/391941752 conn(0x7fc5980b7ea0 msgr2=0x7fc598074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:09.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.794+0000 7fc5a038f640 1 -- 172.21.15.63:0/391941752 shutdown_connections 2024-03-25T18:21:09.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.794+0000 7fc5a038f640 1 -- 172.21.15.63:0/391941752 wait complete. 2024-03-25T18:21:09.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:09.794+0000 7fc5a038f640 1 librados: shutdown 2024-03-25T18:21:09.807 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:16.614 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:16.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.729+0000 7f3dc0d3b640 10 monclient: get_monmap_and_config 2024-03-25T18:21:16.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.729+0000 7f3dc0d3b640 10 monclient: build_initial_monmap 2024-03-25T18:21:16.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.729+0000 7f3dc0d3b640 10 monclient: monmap: 2024-03-25T18:21:16.731 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:21:16.731 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:16.731 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:21:16.730574+0000 2024-03-25T18:21:16.731 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:21:16.730574+0000 2024-03-25T18:21:16.731 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:16.731 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:16.731 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:21:16.731 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:21:16.731 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:21:16.731 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:16.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.730+0000 7f3dc0d3b640 1 Processor -- start 2024-03-25T18:21:16.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.730+0000 7f3dc0d3b640 1 -- start start 2024-03-25T18:21:16.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.730+0000 7f3dc0d3b640 10 monclient: init 2024-03-25T18:21:16.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.731+0000 7f3dc0d3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:16.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.731+0000 7f3dc0d3b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:16.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.731+0000 7f3dc0d3b640 10 monclient(hunting): picked mon.noname-c con 0x7f3dbc0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:21:16.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.731+0000 7f3dc0d3b640 10 monclient(hunting): picked mon.noname-a con 0x7f3dbc13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:21:16.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.731+0000 7f3dc0d3b640 10 monclient(hunting): picked mon.noname-b con 0x7f3dbc1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:21:16.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.731+0000 7f3dc0d3b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dbc142520 con 0x7f3dbc13f710 2024-03-25T18:21:16.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.731+0000 7f3dc0d3b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dbc0b28a0 con 0x7f3dbc0a98f0 2024-03-25T18:21:16.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.731+0000 7f3dc0d3b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dbc0b2a30 con 0x7f3dbc1430c0 2024-03-25T18:21:16.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.731+0000 7f3dc0d3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:16.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.731+0000 7f3dc0d3b640 10 monclient(hunting): authenticate will time out at 7970.917572s 2024-03-25T18:21:16.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.746+0000 7f3dbad9d640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f3dbc1430c0 0x7f3dbc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:57848/0 (socket says 172.21.15.63:57848) 2024-03-25T18:21:16.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.746+0000 7f3dbad9d640 1 -- 172.21.15.63:0/3123932291 learned_addr learned my addr 172.21.15.63:0/3123932291 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:21:16.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.758+0000 7f3db959a640 1 -- 172.21.15.63:0/3123932291 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119454347 0 0) 0x7f3dbc0b2a30 con 0x7f3dbc1430c0 2024-03-25T18:21:16.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.758+0000 7f3db959a640 10 monclient(hunting): _negotiate 2024-03-25T18:21:16.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.758+0000 7f3db959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:16.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.758+0000 7f3db959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:16.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.758+0000 7f3db959a640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:21:16.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.758+0000 7f3db959a640 1 -- 172.21.15.63:0/3123932291 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da4003520 con 0x7f3dbc1430c0 2024-03-25T18:21:16.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.759+0000 7f3db959a640 1 -- 172.21.15.63:0/3123932291 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156519353 0 0) 0x7f3dbc142520 con 0x7f3dbc13f710 2024-03-25T18:21:16.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.759+0000 7f3db959a640 10 monclient(hunting): _negotiate 2024-03-25T18:21:16.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.759+0000 7f3db959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:16.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.759+0000 7f3db959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:16.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.759+0000 7f3db959a640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:21:16.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.759+0000 7f3db959a640 1 -- 172.21.15.63:0/3123932291 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dbc0b2a30 con 0x7f3dbc13f710 2024-03-25T18:21:16.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.761+0000 7f3db959a640 1 -- 172.21.15.63:0/3123932291 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781200600 0 0) 0x7f3dbc0b28a0 con 0x7f3dbc0a98f0 2024-03-25T18:21:16.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.761+0000 7f3db959a640 10 monclient(hunting): _negotiate 2024-03-25T18:21:16.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.761+0000 7f3db959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:16.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.761+0000 7f3db959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:16.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.761+0000 7f3db959a640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:21:16.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.761+0000 7f3db959a640 1 -- 172.21.15.63:0/3123932291 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dbc142520 con 0x7f3dbc0a98f0 2024-03-25T18:21:16.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3db959a640 1 -- 172.21.15.63:0/3123932291 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3854946609 0 0) 0x7f3da4003520 con 0x7f3dbc1430c0 2024-03-25T18:21:16.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3db959a640 1 -- 172.21.15.63:0/3123932291 >> v1:172.21.15.81:6789/0 conn(0x7f3dbc0a98f0 legacy=0x7f3dbc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:16.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3db959a640 1 -- 172.21.15.63:0/3123932291 >> v1:172.21.15.63:6789/0 conn(0x7f3dbc13f710 legacy=0x7f3dbc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:16.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3db959a640 10 monclient: _finish_hunting 0 2024-03-25T18:21:16.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3db959a640 1 monclient: found mon.noname-b 2024-03-25T18:21:16.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3db959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:16.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3db959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:21:16.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3db959a640 1 -- 172.21.15.63:0/3123932291 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dbc0b44c0 con 0x7f3dbc1430c0 2024-03-25T18:21:16.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3db959a640 10 monclient: _finish_auth 0 2024-03-25T18:21:16.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3db959a640 10 monclient: _check_auth_tickets 2024-03-25T18:21:16.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3db959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:16.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3db959a640 1 -- 172.21.15.63:0/3123932291 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3db4001a10 con 0x7f3dbc1430c0 2024-03-25T18:21:16.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3db959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:16.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3db959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:16.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3db959a640 10 monclient: dump: 2024-03-25T18:21:16.779 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:16.779 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:16.779 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:16.779 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:16.779 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:16.779 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:16.779 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:16.779 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:16.779 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:16.779 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:16.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3dc0d3b640 5 monclient: authenticate success, global_id 4601 2024-03-25T18:21:16.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.776+0000 7f3dc0d3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:16.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.789+0000 7f3db959a640 1 -- 172.21.15.63:0/3123932291 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3db4001b70 con 0x7f3dbc1430c0 2024-03-25T18:21:16.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.789+0000 7f3db959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:16.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.789+0000 7f3db959a640 1 -- 172.21.15.63:0/3123932291 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3db4001f80 con 0x7f3dbc1430c0 2024-03-25T18:21:16.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.789+0000 7f3dc0d3b640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:16.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.789+0000 7f3dc0d3b640 4 set_mon_vals no callback set 2024-03-25T18:21:16.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.789+0000 7f3db959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:16.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.789+0000 7f3db959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:16.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.789+0000 7f3db959a640 10 monclient: dump: 2024-03-25T18:21:16.791 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:16.791 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:16.791 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:16.791 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:16.791 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:16.791 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:16.791 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:16.791 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:16.791 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:16.791 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:16.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.790+0000 7f3dc0d3b640 10 monclient: shutdown 2024-03-25T18:21:16.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.790+0000 7f3dc0d3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:16.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.790+0000 7f3dc0d3b640 1 -- 172.21.15.63:0/3123932291 >> v1:172.21.15.63:6790/0 conn(0x7f3dbc1430c0 legacy=0x7f3dbc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:16.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.790+0000 7f3dc0d3b640 1 -- 172.21.15.63:0/3123932291 shutdown_connections 2024-03-25T18:21:16.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.790+0000 7f3dc0d3b640 1 -- 172.21.15.63:0/3123932291 >> 172.21.15.63:0/3123932291 conn(0x7f3dbc0b7ea0 msgr2=0x7f3dbc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:16.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.790+0000 7f3dc0d3b640 1 -- 172.21.15.63:0/3123932291 shutdown_connections 2024-03-25T18:21:16.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.790+0000 7f3dc0d3b640 1 -- 172.21.15.63:0/3123932291 wait complete. 2024-03-25T18:21:16.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.791+0000 7f3dc0d3b640 10 monclient: build_initial_monmap 2024-03-25T18:21:16.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.791+0000 7f3dc0d3b640 10 monclient: monmap: 2024-03-25T18:21:16.793 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:21:16.793 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:16.793 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:21:16.793 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:21:16.793 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:16.793 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:21:16.793 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:21:16.793 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:21:16.794 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:21:16.794 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:16.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.791+0000 7f3dc0d3b640 1 librados: starting msgr at 2024-03-25T18:21:16.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.791+0000 7f3dc0d3b640 1 librados: starting objecter 2024-03-25T18:21:16.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.791+0000 7f3dc0d3b640 1 Processor -- start 2024-03-25T18:21:16.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.791+0000 7f3dc0d3b640 1 -- start start 2024-03-25T18:21:16.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.791+0000 7f3dc0d3b640 1 librados: setting wanted keys 2024-03-25T18:21:16.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.791+0000 7f3dc0d3b640 1 librados: calling monclient init 2024-03-25T18:21:16.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.791+0000 7f3dc0d3b640 10 monclient: init 2024-03-25T18:21:16.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.792+0000 7f3dc0d3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:16.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.792+0000 7f3dc0d3b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:16.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.792+0000 7f3dc0d3b640 10 monclient(hunting): picked mon.noname-c con 0x7f3dbc0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:21:16.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.792+0000 7f3dc0d3b640 10 monclient(hunting): picked mon.noname-a con 0x7f3dbc13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:21:16.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.792+0000 7f3dc0d3b640 10 monclient(hunting): picked mon.noname-b con 0x7f3dbc1430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:21:16.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.792+0000 7f3dc0d3b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dbc1d8340 con 0x7f3dbc13f710 2024-03-25T18:21:16.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.792+0000 7f3dc0d3b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dbc1d7710 con 0x7f3dbc1430c0 2024-03-25T18:21:16.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.792+0000 7f3dc0d3b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dbc138e90 con 0x7f3dbc0a98f0 2024-03-25T18:21:16.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.792+0000 7f3dc0d3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:16.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.792+0000 7f3dc0d3b640 10 monclient(hunting): authenticate will time out at 7970.978670s 2024-03-25T18:21:16.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.805+0000 7f3db9d9b640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f3dbc13f710 0x7f3dbc138780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:49366/0 (socket says 172.21.15.63:49366) 2024-03-25T18:21:16.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.805+0000 7f3db9d9b640 1 -- 172.21.15.63:0/31058864 learned_addr learned my addr 172.21.15.63:0/31058864 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:21:16.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.806+0000 7f3daaffd640 10 client.?.objecter ms_handle_connect 0x7f3dbc13f710 2024-03-25T18:21:16.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.806+0000 7f3daaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:16.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.809+0000 7f3daaffd640 10 client.?.objecter ms_handle_connect 0x7f3dbc1430c0 2024-03-25T18:21:16.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.809+0000 7f3daaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:16.814 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.813+0000 7f3daaffd640 10 client.?.objecter ms_handle_connect 0x7f3dbc0a98f0 2024-03-25T18:21:16.814 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.813+0000 7f3daaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:16.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.821+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987035051 0 0) 0x7f3dbc1d8340 con 0x7f3dbc13f710 2024-03-25T18:21:16.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.821+0000 7f3daaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:16.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.821+0000 7f3daaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:16.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.821+0000 7f3daaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:16.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.821+0000 7f3daaffd640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:21:16.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.821+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d9c004eb0 con 0x7f3dbc13f710 2024-03-25T18:21:16.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.825+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052153356 0 0) 0x7f3dbc1d7710 con 0x7f3dbc1430c0 2024-03-25T18:21:16.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.825+0000 7f3daaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:16.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.825+0000 7f3daaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:16.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.825+0000 7f3daaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:16.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.825+0000 7f3daaffd640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:21:16.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.825+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dbc1d8340 con 0x7f3dbc1430c0 2024-03-25T18:21:16.832 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.831+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 369093530 0 0) 0x7f3d9c004eb0 con 0x7f3dbc13f710 2024-03-25T18:21:16.832 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.831+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3dbc1d7710 con 0x7f3dbc13f710 2024-03-25T18:21:16.832 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.831+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3db0001a30 con 0x7f3dbc13f710 2024-03-25T18:21:16.832 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.831+0000 7f3daaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:16.833 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.831+0000 7f3daaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:16.833 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.831+0000 7f3daaffd640 10 monclient(hunting): dump: 2024-03-25T18:21:16.833 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:16.833 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:16.833 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:16.833 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:16.833 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:16.833 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:16.833 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:16.833 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:16.833 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:16.833 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:16.833 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.831+0000 7f3daaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:16.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2270470354 0 0) 0x7f3dbc1d7710 con 0x7f3dbc13f710 2024-03-25T18:21:16.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 >> v1:172.21.15.63:6790/0 conn(0x7f3dbc0a98f0 legacy=0x7f3dbc1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:16.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 >> v1:172.21.15.81:6789/0 conn(0x7f3dbc1430c0 legacy=0x7f3dbc133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:16.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3daaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:16.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3daaffd640 1 monclient: found mon.a 2024-03-25T18:21:16.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3daaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:16.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3daaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:21:16.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dbc139020 con 0x7f3dbc13f710 2024-03-25T18:21:16.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3daaffd640 10 monclient: _finish_auth 0 2024-03-25T18:21:16.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3daaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:16.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3daaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:16.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3dc0d3b640 5 monclient: authenticate success, global_id 4596 2024-03-25T18:21:16.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3dc0d3b640 10 monclient: _renew_subs 2024-03-25T18:21:16.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3dc0d3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:21:16.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3dc0d3b640 1 -- 172.21.15.63:0/31058864 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dbc137f30 con 0x7f3dbc13f710 2024-03-25T18:21:16.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3dc0d3b640 10 client.4596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:16.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3dc0d3b640 10 monclient: _renew_subs 2024-03-25T18:21:16.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3dc0d3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:21:16.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3dc0d3b640 1 -- 172.21.15.63:0/31058864 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dbc1352b0 con 0x7f3dbc13f710 2024-03-25T18:21:16.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.837+0000 7f3dc0d3b640 1 librados: init done 2024-03-25T18:21:16.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.838+0000 7f3dc0d3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:16.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.838+0000 7f3dc0d3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:16.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.838+0000 7f3dc0d3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:21:16.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:16.838+0000 7f3dc0d3b640 1 -- 172.21.15.63:0/31058864 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dbc1d7950 con 0x7f3dbc13f710 2024-03-25T18:21:17.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.044+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3db0001c90 con 0x7f3dbc13f710 2024-03-25T18:21:17.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.044+0000 7f3daaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:17.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.044+0000 7f3db8d99640 4 set_mon_vals no callback set 2024-03-25T18:21:17.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.139+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f3db00038c0 con 0x7f3dbc13f710 2024-03-25T18:21:17.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.139+0000 7f3daaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:17.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.139+0000 7f3daaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:17.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.139+0000 7f3daaffd640 10 monclient: dump: 2024-03-25T18:21:17.140 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:17.141 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:17.141 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:17.141 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:17.141 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:17.141 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:17.141 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:17.141 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:17.141 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:17.141 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:17.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.140+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f3db0003aa0 con 0x7f3dbc13f710 2024-03-25T18:21:17.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.140+0000 7f3dba59c640 10 monclient: get_auth_request con 0x7f3d9c06dd10 auth_method 0 2024-03-25T18:21:17.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.141+0000 7f3daaffd640 10 client.4596.objecter ms_handle_connect 0x7f3d9c06dd10 2024-03-25T18:21:17.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.146+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f3db0004460 con 0x7f3dbc13f710 2024-03-25T18:21:17.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.146+0000 7f3daaffd640 10 client.4596.objecter ms_dispatch 0x7f3dbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:17.147 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.146+0000 7f3daaffd640 3 client.4596.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:17.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.146+0000 7f3daaffd640 3 client.4596.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:17.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.146+0000 7f3daaffd640 20 client.4596.objecter dump_active .. 0 homeless 2024-03-25T18:21:17.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.146+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3db00502a0 con 0x7f3dbc13f710 2024-03-25T18:21:17.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.146+0000 7f3daaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:17.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.146+0000 7f3daaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:17.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.397+0000 7f3dc0d3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:17.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.397+0000 7f3dc0d3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:17.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.397+0000 7f3dc0d3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:21:17.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.397+0000 7f3dc0d3b640 1 -- 172.21.15.63:0/31058864 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3dbc138230 con 0x7f3dbc13f710 2024-03-25T18:21:17.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.415+0000 7f3daaffd640 1 -- 172.21.15.63:0/31058864 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f3db00521d0 con 0x7f3dbc13f710 2024-03-25T18:21:17.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.415+0000 7f3daaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:17.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.415+0000 7f3daaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:17.417 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:21:17.417 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:17.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.417+0000 7f3dc0d3b640 10 librados: watch_flush enter 2024-03-25T18:21:17.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.417+0000 7f3dc0d3b640 10 librados: watch_flush exit 2024-03-25T18:21:17.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.417+0000 7f3dc0d3b640 20 client.4596.objecter shutdown clearing up homeless session... 2024-03-25T18:21:17.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.417+0000 7f3dc0d3b640 10 client.4596.objecter successfully canceled tick 2024-03-25T18:21:17.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.417+0000 7f3dc0d3b640 1 -- 172.21.15.63:0/31058864 >> v1:172.21.15.81:6800/3640340873 conn(0x7f3d9c06dd10 legacy=0x7f3d9c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:17.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.417+0000 7f3dc0d3b640 10 monclient: shutdown 2024-03-25T18:21:17.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.417+0000 7f3dc0d3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:17.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.418+0000 7f3dc0d3b640 1 -- 172.21.15.63:0/31058864 >> v1:172.21.15.63:6789/0 conn(0x7f3dbc13f710 legacy=0x7f3dbc138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:17.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.418+0000 7f3dc0d3b640 1 -- 172.21.15.63:0/31058864 shutdown_connections 2024-03-25T18:21:17.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.418+0000 7f3dc0d3b640 1 -- 172.21.15.63:0/31058864 >> 172.21.15.63:0/31058864 conn(0x7f3dbc0b7ea0 msgr2=0x7f3dbc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:17.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.418+0000 7f3dc0d3b640 1 -- 172.21.15.63:0/31058864 shutdown_connections 2024-03-25T18:21:17.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.418+0000 7f3dc0d3b640 1 -- 172.21.15.63:0/31058864 wait complete. 2024-03-25T18:21:17.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:17.418+0000 7f3dc0d3b640 1 librados: shutdown 2024-03-25T18:21:17.430 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:24.237 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:24.352 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.351+0000 7fdbe617b640 10 monclient: get_monmap_and_config 2024-03-25T18:21:24.352 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.351+0000 7fdbe617b640 10 monclient: build_initial_monmap 2024-03-25T18:21:24.352 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.351+0000 7fdbe617b640 10 monclient: monmap: 2024-03-25T18:21:24.352 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:21:24.352 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:24.352 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:21:24.352260+0000 2024-03-25T18:21:24.352 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:21:24.352260+0000 2024-03-25T18:21:24.352 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:24.353 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:24.353 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:21:24.353 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:21:24.353 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:21:24.353 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:24.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.351+0000 7fdbe617b640 1 Processor -- start 2024-03-25T18:21:24.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.351+0000 7fdbe617b640 1 -- start start 2024-03-25T18:21:24.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.351+0000 7fdbe617b640 10 monclient: init 2024-03-25T18:21:24.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.352+0000 7fdbe617b640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:24.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.352+0000 7fdbe617b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:24.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.352+0000 7fdbe617b640 10 monclient(hunting): picked mon.noname-b con 0x7fdbe00b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T18:21:24.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.352+0000 7fdbe617b640 10 monclient(hunting): picked mon.noname-c con 0x7fdbe00b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:21:24.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.352+0000 7fdbe617b640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe014be90 addr v1:172.21.15.63:6789/0 2024-03-25T18:21:24.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.352+0000 7fdbe617b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe00af940 con 0x7fdbe014be90 2024-03-25T18:21:24.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.352+0000 7fdbe617b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe00b0a70 con 0x7fdbe00b1730 2024-03-25T18:21:24.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.352+0000 7fdbe617b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe00b0d50 con 0x7fdbe00b4d60 2024-03-25T18:21:24.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.352+0000 7fdbe617b640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:24.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.352+0000 7fdbe617b640 10 monclient(hunting): authenticate will time out at 7978.538980s 2024-03-25T18:21:24.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.367+0000 7fdbdf7fe640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fdbe00b4d60 0x7fdbe013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:43720/0 (socket says 172.21.15.63:43720) 2024-03-25T18:21:24.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.367+0000 7fdbdf7fe640 1 -- 172.21.15.63:0/3342855598 learned_addr learned my addr 172.21.15.63:0/3342855598 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:21:24.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.378+0000 7fdbde7fc640 1 -- 172.21.15.63:0/3342855598 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3432412126 0 0) 0x7fdbe00b0d50 con 0x7fdbe00b4d60 2024-03-25T18:21:24.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.378+0000 7fdbde7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:24.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.378+0000 7fdbde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:24.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.378+0000 7fdbde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:24.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.378+0000 7fdbde7fc640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:21:24.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.378+0000 7fdbde7fc640 1 -- 172.21.15.63:0/3342855598 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc8003520 con 0x7fdbe00b4d60 2024-03-25T18:21:24.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbde7fc640 1 -- 172.21.15.63:0/3342855598 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2825750917 0 0) 0x7fdbc8003520 con 0x7fdbe00b4d60 2024-03-25T18:21:24.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbde7fc640 1 -- 172.21.15.63:0/3342855598 >> v1:172.21.15.81:6789/0 conn(0x7fdbe00b1730 legacy=0x7fdbe00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:24.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbde7fc640 1 -- 172.21.15.63:0/3342855598 >> v1:172.21.15.63:6789/0 conn(0x7fdbe014be90 legacy=0x7fdbe014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:24.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbde7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:24.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbde7fc640 1 monclient: found mon.noname-b 2024-03-25T18:21:24.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:24.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbde7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:21:24.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbde7fc640 1 -- 172.21.15.63:0/3342855598 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe00b1030 con 0x7fdbe00b4d60 2024-03-25T18:21:24.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbde7fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:24.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbde7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:24.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:24.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbde7fc640 1 -- 172.21.15.63:0/3342855598 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fdbd0001690 con 0x7fdbe00b4d60 2024-03-25T18:21:24.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbe617b640 5 monclient: authenticate success, global_id 4607 2024-03-25T18:21:24.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbe617b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:24.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:24.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbde7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:24.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbde7fc640 10 monclient: dump: 2024-03-25T18:21:24.401 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:24.401 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:24.401 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:24.401 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:24.401 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:24.401 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:24.401 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:24.401 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:24.401 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:24.401 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:24.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.398+0000 7fdbe617b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:24.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.423+0000 7fdbde7fc640 1 -- 172.21.15.63:0/3342855598 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbd00017f0 con 0x7fdbe00b4d60 2024-03-25T18:21:24.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.423+0000 7fdbde7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:24.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.423+0000 7fdbde7fc640 1 -- 172.21.15.63:0/3342855598 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fdbd0001c40 con 0x7fdbe00b4d60 2024-03-25T18:21:24.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.423+0000 7fdbe617b640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:24.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.423+0000 7fdbe617b640 4 set_mon_vals no callback set 2024-03-25T18:21:24.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.423+0000 7fdbde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:24.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.423+0000 7fdbde7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:24.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.423+0000 7fdbde7fc640 10 monclient: dump: 2024-03-25T18:21:24.425 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:24.425 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:24.425 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:24.425 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:24.425 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:24.425 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:24.425 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:24.425 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:24.425 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:24.425 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:24.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.424+0000 7fdbe617b640 10 monclient: shutdown 2024-03-25T18:21:24.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.424+0000 7fdbe617b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:24.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.424+0000 7fdbe617b640 1 -- 172.21.15.63:0/3342855598 >> v1:172.21.15.63:6790/0 conn(0x7fdbe00b4d60 legacy=0x7fdbe013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:24.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.424+0000 7fdbe617b640 1 -- 172.21.15.63:0/3342855598 shutdown_connections 2024-03-25T18:21:24.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.424+0000 7fdbe617b640 1 -- 172.21.15.63:0/3342855598 >> 172.21.15.63:0/3342855598 conn(0x7fdbe00617a0 msgr2=0x7fdbe0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:24.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.424+0000 7fdbe617b640 1 -- 172.21.15.63:0/3342855598 shutdown_connections 2024-03-25T18:21:24.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.424+0000 7fdbe617b640 1 -- 172.21.15.63:0/3342855598 wait complete. 2024-03-25T18:21:24.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.424+0000 7fdbe617b640 10 monclient: build_initial_monmap 2024-03-25T18:21:24.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.425+0000 7fdbe617b640 10 monclient: monmap: 2024-03-25T18:21:24.429 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:21:24.429 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:24.429 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:21:24.429 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:21:24.429 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:24.429 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:21:24.429 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:21:24.429 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:21:24.429 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:21:24.429 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:24.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.425+0000 7fdbe617b640 1 librados: starting msgr at 2024-03-25T18:21:24.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.425+0000 7fdbe617b640 1 librados: starting objecter 2024-03-25T18:21:24.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.425+0000 7fdbe617b640 1 Processor -- start 2024-03-25T18:21:24.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.425+0000 7fdbe617b640 1 -- start start 2024-03-25T18:21:24.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.425+0000 7fdbe617b640 1 librados: setting wanted keys 2024-03-25T18:21:24.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.425+0000 7fdbe617b640 1 librados: calling monclient init 2024-03-25T18:21:24.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.425+0000 7fdbe617b640 10 monclient: init 2024-03-25T18:21:24.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.426+0000 7fdbe617b640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:24.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.426+0000 7fdbe617b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:24.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.426+0000 7fdbe617b640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe00b1730 addr v1:172.21.15.63:6789/0 2024-03-25T18:21:24.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.426+0000 7fdbe617b640 10 monclient(hunting): picked mon.noname-b con 0x7fdbe00b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T18:21:24.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.426+0000 7fdbe617b640 10 monclient(hunting): picked mon.noname-c con 0x7fdbe014be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:21:24.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.426+0000 7fdbe617b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe01361d0 con 0x7fdbe00b1730 2024-03-25T18:21:24.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.426+0000 7fdbe617b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe0136c40 con 0x7fdbe00b4d60 2024-03-25T18:21:24.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.426+0000 7fdbe617b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe0136f20 con 0x7fdbe014be90 2024-03-25T18:21:24.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.426+0000 7fdbe617b640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:24.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.426+0000 7fdbe617b640 10 monclient(hunting): authenticate will time out at 7978.613145s 2024-03-25T18:21:24.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.434+0000 7fdbdeffd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fdbe00b4d60 0x7fdbe012cdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:40626/0 (socket says 172.21.15.63:40626) 2024-03-25T18:21:24.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.434+0000 7fdbdeffd640 1 -- 172.21.15.63:0/3621929906 learned_addr learned my addr 172.21.15.63:0/3621929906 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:21:24.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.434+0000 7fdbc7fff640 10 client.?.objecter ms_handle_connect 0x7fdbe00b4d60 2024-03-25T18:21:24.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.434+0000 7fdbc7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:24.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.437+0000 7fdbc7fff640 1 -- 172.21.15.63:0/3621929906 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095853473 0 0) 0x7fdbe0136c40 con 0x7fdbe00b4d60 2024-03-25T18:21:24.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.437+0000 7fdbc7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:24.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.437+0000 7fdbc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:24.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.437+0000 7fdbc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:24.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.437+0000 7fdbc7fff640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:21:24.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.437+0000 7fdbc7fff640 1 -- 172.21.15.63:0/3621929906 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbcc004eb0 con 0x7fdbe00b4d60 2024-03-25T18:21:24.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.449+0000 7fdbc7fff640 1 -- 172.21.15.63:0/3621929906 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 362733471 0 0) 0x7fdbcc004eb0 con 0x7fdbe00b4d60 2024-03-25T18:21:24.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.449+0000 7fdbc7fff640 1 -- 172.21.15.63:0/3621929906 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbe0136c40 con 0x7fdbe00b4d60 2024-03-25T18:21:24.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.449+0000 7fdbc7fff640 1 -- 172.21.15.63:0/3621929906 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fdbd40019d0 con 0x7fdbe00b4d60 2024-03-25T18:21:24.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.449+0000 7fdbc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:24.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.449+0000 7fdbc7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:24.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.449+0000 7fdbc7fff640 10 monclient(hunting): dump: 2024-03-25T18:21:24.451 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:24.451 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:24.451 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:24.451 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:24.451 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:24.451 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:24.451 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:24.451 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:24.451 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:24.451 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:24.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.449+0000 7fdbc7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:24.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.452+0000 7fdbc7fff640 10 client.?.objecter ms_handle_connect 0x7fdbe014be90 2024-03-25T18:21:24.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.452+0000 7fdbc7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:24.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.463+0000 7fdbc7fff640 10 client.?.objecter ms_handle_connect 0x7fdbe00b1730 2024-03-25T18:21:24.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.463+0000 7fdbc7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:24.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.464+0000 7fdbc7fff640 1 -- 172.21.15.63:0/3621929906 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1505715132 0 0) 0x7fdbe0136c40 con 0x7fdbe00b4d60 2024-03-25T18:21:24.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.464+0000 7fdbc7fff640 1 -- 172.21.15.63:0/3621929906 >> v1:172.21.15.63:6790/0 conn(0x7fdbe014be90 legacy=0x7fdbe01e33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:24.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.464+0000 7fdbc7fff640 1 -- 172.21.15.63:0/3621929906 >> v1:172.21.15.63:6789/0 conn(0x7fdbe00b1730 legacy=0x7fdbe012e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:24.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.464+0000 7fdbc7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:24.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.464+0000 7fdbc7fff640 1 monclient: found mon.b 2024-03-25T18:21:24.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.464+0000 7fdbc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:24.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.464+0000 7fdbc7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:24.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.464+0000 7fdbc7fff640 1 -- 172.21.15.63:0/3621929906 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe0137200 con 0x7fdbe00b4d60 2024-03-25T18:21:24.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.464+0000 7fdbc7fff640 10 monclient: _finish_auth 0 2024-03-25T18:21:24.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.464+0000 7fdbc7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:24.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.464+0000 7fdbc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:24.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.464+0000 7fdbe617b640 5 monclient: authenticate success, global_id 4636 2024-03-25T18:21:24.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.464+0000 7fdbe617b640 10 monclient: _renew_subs 2024-03-25T18:21:24.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.464+0000 7fdbe617b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:24.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.464+0000 7fdbe617b640 1 -- 172.21.15.63:0/3621929906 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe0135cb0 con 0x7fdbe00b4d60 2024-03-25T18:21:24.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.465+0000 7fdbe617b640 10 client.4636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:24.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.465+0000 7fdbe617b640 10 monclient: _renew_subs 2024-03-25T18:21:24.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.465+0000 7fdbe617b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:24.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.465+0000 7fdbe617b640 1 -- 172.21.15.63:0/3621929906 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe013e930 con 0x7fdbe00b4d60 2024-03-25T18:21:24.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.465+0000 7fdbe617b640 1 librados: init done 2024-03-25T18:21:24.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.466+0000 7fdbe617b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:24.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.466+0000 7fdbe617b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:24.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.466+0000 7fdbe617b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:24.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.466+0000 7fdbe617b640 1 -- 172.21.15.63:0/3621929906 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe01401f0 con 0x7fdbe00b4d60 2024-03-25T18:21:24.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.732+0000 7fdbc7fff640 1 -- 172.21.15.63:0/3621929906 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbd4001cd0 con 0x7fdbe00b4d60 2024-03-25T18:21:24.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.732+0000 7fdbc7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:24.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.732+0000 7fdbc7fff640 1 -- 172.21.15.63:0/3621929906 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fdbd4003a60 con 0x7fdbe00b4d60 2024-03-25T18:21:24.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.732+0000 7fdbc7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:24.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.732+0000 7fdbc7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:24.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.732+0000 7fdbc7fff640 10 monclient: dump: 2024-03-25T18:21:24.734 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:24.734 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:24.734 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:24.734 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:24.734 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:24.734 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:24.734 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:24.734 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:24.734 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:24.734 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:24.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.732+0000 7fdbddffb640 4 set_mon_vals no callback set 2024-03-25T18:21:24.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.732+0000 7fdbc7fff640 1 -- 172.21.15.63:0/3621929906 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fdbd4003c80 con 0x7fdbe00b4d60 2024-03-25T18:21:24.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.733+0000 7fdbdf7fe640 10 monclient: get_auth_request con 0x7fdbcc06dcf0 auth_method 0 2024-03-25T18:21:24.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.733+0000 7fdbc7fff640 10 client.4636.objecter ms_handle_connect 0x7fdbcc06dcf0 2024-03-25T18:21:24.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.739+0000 7fdbc7fff640 1 -- 172.21.15.63:0/3621929906 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fdbd4054890 con 0x7fdbe00b4d60 2024-03-25T18:21:24.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.739+0000 7fdbc7fff640 10 client.4636.objecter ms_dispatch 0x7fdbe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:24.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.739+0000 7fdbc7fff640 3 client.4636.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:24.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.739+0000 7fdbc7fff640 3 client.4636.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:24.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.739+0000 7fdbc7fff640 20 client.4636.objecter dump_active .. 0 homeless 2024-03-25T18:21:24.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.740+0000 7fdbc7fff640 1 -- 172.21.15.63:0/3621929906 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdbd40503a0 con 0x7fdbe00b4d60 2024-03-25T18:21:24.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.740+0000 7fdbc7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:24.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.740+0000 7fdbc7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:24.941 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:06.633 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:24.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.943+0000 7fdbe617b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:24.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.943+0000 7fdbe617b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:24.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.943+0000 7fdbe617b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:24.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:24.943+0000 7fdbe617b640 1 -- 172.21.15.63:0/3621929906 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbe00afbd0 con 0x7fdbe00b4d60 2024-03-25T18:21:24.947 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:06.640 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:24.969 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:06.661 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:25.017 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:06.709 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:25.017 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:09:06.678 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:25.026 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:06.718 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:25.026 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:09:06.687 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:25.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:25.030+0000 7fdbc7fff640 1 -- 172.21.15.63:0/3621929906 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fdbd40522d0 con 0x7fdbe00b4d60 2024-03-25T18:21:25.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:25.030+0000 7fdbc7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:25.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:25.030+0000 7fdbc7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:25.031 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:21:25.031 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:25.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:25.032+0000 7fdbe617b640 10 librados: watch_flush enter 2024-03-25T18:21:25.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:25.032+0000 7fdbe617b640 10 librados: watch_flush exit 2024-03-25T18:21:25.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:25.032+0000 7fdbe617b640 20 client.4636.objecter shutdown clearing up homeless session... 2024-03-25T18:21:25.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:25.032+0000 7fdbe617b640 10 client.4636.objecter successfully canceled tick 2024-03-25T18:21:25.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:25.032+0000 7fdbe617b640 1 -- 172.21.15.63:0/3621929906 >> v1:172.21.15.81:6800/3640340873 conn(0x7fdbcc06dcf0 legacy=0x7fdbcc0701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:25.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:25.032+0000 7fdbe617b640 10 monclient: shutdown 2024-03-25T18:21:25.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:25.032+0000 7fdbe617b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:25.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:25.032+0000 7fdbe617b640 1 -- 172.21.15.63:0/3621929906 >> v1:172.21.15.81:6789/0 conn(0x7fdbe00b4d60 legacy=0x7fdbe012cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:25.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:25.032+0000 7fdbe617b640 1 -- 172.21.15.63:0/3621929906 shutdown_connections 2024-03-25T18:21:25.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:25.032+0000 7fdbe617b640 1 -- 172.21.15.63:0/3621929906 >> 172.21.15.63:0/3621929906 conn(0x7fdbe00617a0 msgr2=0x7fdbe00b6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:25.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:25.032+0000 7fdbe617b640 1 -- 172.21.15.63:0/3621929906 shutdown_connections 2024-03-25T18:21:25.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:25.032+0000 7fdbe617b640 1 -- 172.21.15.63:0/3621929906 wait complete. 2024-03-25T18:21:25.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:25.032+0000 7fdbe617b640 1 librados: shutdown 2024-03-25T18:21:25.045 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:25.048 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:09:06.708 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:25.062 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:06.754 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:31.851 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:31.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.961+0000 7fddbfb67640 10 monclient: get_monmap_and_config 2024-03-25T18:21:31.962 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.961+0000 7fddbfb67640 10 monclient: build_initial_monmap 2024-03-25T18:21:31.962 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.961+0000 7fddbfb67640 10 monclient: monmap: 2024-03-25T18:21:31.962 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:21:31.962 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:31.962 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:21:31.961819+0000 2024-03-25T18:21:31.962 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:21:31.961819+0000 2024-03-25T18:21:31.962 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:31.962 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:31.962 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:21:31.962 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:21:31.962 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:21:31.962 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:31.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.962+0000 7fddbfb67640 1 Processor -- start 2024-03-25T18:21:31.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.962+0000 7fddbfb67640 1 -- start start 2024-03-25T18:21:31.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.962+0000 7fddbfb67640 10 monclient: init 2024-03-25T18:21:31.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.963+0000 7fddbfb67640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:31.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.963+0000 7fddbfb67640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:31.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.963+0000 7fddbfb67640 10 monclient(hunting): picked mon.noname-a con 0x7fddb80a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:21:31.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.963+0000 7fddbfb67640 10 monclient(hunting): picked mon.noname-b con 0x7fddb813f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:21:31.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.963+0000 7fddbfb67640 10 monclient(hunting): picked mon.noname-c con 0x7fddb81430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:21:31.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.963+0000 7fddbfb67640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb8142520 con 0x7fddb80a98f0 2024-03-25T18:21:31.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.963+0000 7fddbfb67640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb80b28a0 con 0x7fddb81430c0 2024-03-25T18:21:31.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.963+0000 7fddbfb67640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb80b2a30 con 0x7fddb813f710 2024-03-25T18:21:31.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.963+0000 7fddbfb67640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:31.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.963+0000 7fddbfb67640 10 monclient(hunting): authenticate will time out at 7986.148634s 2024-03-25T18:21:31.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.979+0000 7fddbd102640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fddb813f710 0x7fddb81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:34474/0 (socket says 172.21.15.63:34474) 2024-03-25T18:21:31.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:31.979+0000 7fddbd102640 1 -- 172.21.15.63:0/1722039874 learned_addr learned my addr 172.21.15.63:0/1722039874 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:21:32.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.001+0000 7fddbc901640 1 -- 172.21.15.63:0/1722039874 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610820816 0 0) 0x7fddb80b2a30 con 0x7fddb813f710 2024-03-25T18:21:32.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.001+0000 7fddbc901640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.001+0000 7fddbc901640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.001+0000 7fddbc901640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.001+0000 7fddbc901640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:21:32.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.001+0000 7fddbc901640 1 -- 172.21.15.63:0/1722039874 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda4003520 con 0x7fddb813f710 2024-03-25T18:21:32.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.002+0000 7fddbc901640 1 -- 172.21.15.63:0/1722039874 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804920618 0 0) 0x7fddb80b28a0 con 0x7fddb81430c0 2024-03-25T18:21:32.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.002+0000 7fddbc901640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.002+0000 7fddbc901640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.002+0000 7fddbc901640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.002+0000 7fddbc901640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:21:32.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.002+0000 7fddbc901640 1 -- 172.21.15.63:0/1722039874 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb80b2a30 con 0x7fddb81430c0 2024-03-25T18:21:32.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.009+0000 7fddbc901640 1 -- 172.21.15.63:0/1722039874 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035290950 0 0) 0x7fddb8142520 con 0x7fddb80a98f0 2024-03-25T18:21:32.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.009+0000 7fddbc901640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.009+0000 7fddbc901640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.009+0000 7fddbc901640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.009+0000 7fddbc901640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:21:32.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.009+0000 7fddbc901640 1 -- 172.21.15.63:0/1722039874 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb80b28a0 con 0x7fddb80a98f0 2024-03-25T18:21:32.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.017+0000 7fddbc901640 1 -- 172.21.15.63:0/1722039874 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1239773093 0 0) 0x7fdda4003520 con 0x7fddb813f710 2024-03-25T18:21:32.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.017+0000 7fddbc901640 1 -- 172.21.15.63:0/1722039874 >> v1:172.21.15.81:6789/0 conn(0x7fddb81430c0 legacy=0x7fddb814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:32.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.017+0000 7fddbc901640 1 -- 172.21.15.63:0/1722039874 >> v1:172.21.15.63:6789/0 conn(0x7fddb80a98f0 legacy=0x7fddb80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:32.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.017+0000 7fddbc901640 10 monclient: _finish_hunting 0 2024-03-25T18:21:32.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.017+0000 7fddbc901640 1 monclient: found mon.noname-b 2024-03-25T18:21:32.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.017+0000 7fddbc901640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:32.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.017+0000 7fddbc901640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:21:32.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.017+0000 7fddbc901640 1 -- 172.21.15.63:0/1722039874 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb80b44c0 con 0x7fddb813f710 2024-03-25T18:21:32.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.017+0000 7fddbc901640 10 monclient: _finish_auth 0 2024-03-25T18:21:32.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.017+0000 7fddbc901640 10 monclient: _check_auth_tickets 2024-03-25T18:21:32.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.017+0000 7fddbc901640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:32.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.018+0000 7fddbfb67640 5 monclient: authenticate success, global_id 4613 2024-03-25T18:21:32.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.018+0000 7fddbfb67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:32.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.017+0000 7fddbc901640 1 -- 172.21.15.63:0/1722039874 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fdda8001690 con 0x7fddb813f710 2024-03-25T18:21:32.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.018+0000 7fddbc901640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:32.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.018+0000 7fddbc901640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:32.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.018+0000 7fddbc901640 10 monclient: dump: 2024-03-25T18:21:32.019 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:32.019 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:32.019 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:32.019 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:32.019 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:32.020 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:32.020 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:32.020 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:32.020 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:32.020 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:32.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.018+0000 7fddbfb67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:32.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.036+0000 7fddbc901640 1 -- 172.21.15.63:0/1722039874 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdda80017f0 con 0x7fddb813f710 2024-03-25T18:21:32.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.036+0000 7fddbc901640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:32.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.036+0000 7fddbc901640 1 -- 172.21.15.63:0/1722039874 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fdda8001c00 con 0x7fddb813f710 2024-03-25T18:21:32.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.036+0000 7fddbfb67640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:32.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.036+0000 7fddbfb67640 4 set_mon_vals no callback set 2024-03-25T18:21:32.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.036+0000 7fddbc901640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:32.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.036+0000 7fddbc901640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:32.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.036+0000 7fddbc901640 10 monclient: dump: 2024-03-25T18:21:32.037 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:32.037 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:32.037 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:32.037 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:32.037 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:32.038 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:32.038 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:32.038 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:32.038 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:32.038 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:32.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.037+0000 7fddbfb67640 10 monclient: shutdown 2024-03-25T18:21:32.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.037+0000 7fddbfb67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:32.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.037+0000 7fddbfb67640 1 -- 172.21.15.63:0/1722039874 >> v1:172.21.15.63:6790/0 conn(0x7fddb813f710 legacy=0x7fddb81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:32.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.037+0000 7fddbfb67640 1 -- 172.21.15.63:0/1722039874 shutdown_connections 2024-03-25T18:21:32.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.037+0000 7fddbfb67640 1 -- 172.21.15.63:0/1722039874 >> 172.21.15.63:0/1722039874 conn(0x7fddb80b7ea0 msgr2=0x7fddb80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:32.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.037+0000 7fddbfb67640 1 -- 172.21.15.63:0/1722039874 shutdown_connections 2024-03-25T18:21:32.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.037+0000 7fddbfb67640 1 -- 172.21.15.63:0/1722039874 wait complete. 2024-03-25T18:21:32.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.038+0000 7fddbfb67640 10 monclient: build_initial_monmap 2024-03-25T18:21:32.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.038+0000 7fddbfb67640 10 monclient: monmap: 2024-03-25T18:21:32.041 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:21:32.041 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:32.041 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:21:32.041 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:21:32.041 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:32.041 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:21:32.041 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:21:32.041 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:21:32.041 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:21:32.041 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:32.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.038+0000 7fddbfb67640 1 librados: starting msgr at 2024-03-25T18:21:32.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.038+0000 7fddbfb67640 1 librados: starting objecter 2024-03-25T18:21:32.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.038+0000 7fddbfb67640 1 Processor -- start 2024-03-25T18:21:32.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.038+0000 7fddbfb67640 1 -- start start 2024-03-25T18:21:32.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.038+0000 7fddbfb67640 1 librados: setting wanted keys 2024-03-25T18:21:32.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.038+0000 7fddbfb67640 1 librados: calling monclient init 2024-03-25T18:21:32.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.038+0000 7fddbfb67640 10 monclient: init 2024-03-25T18:21:32.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.039+0000 7fddbfb67640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:32.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.039+0000 7fddbfb67640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:32.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.039+0000 7fddbfb67640 10 monclient(hunting): picked mon.noname-c con 0x7fddb80a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:21:32.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.039+0000 7fddbfb67640 10 monclient(hunting): picked mon.noname-a con 0x7fddb813f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:21:32.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.039+0000 7fddbfb67640 10 monclient(hunting): picked mon.noname-b con 0x7fddb81430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:21:32.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.039+0000 7fddbfb67640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb80aba00 con 0x7fddb813f710 2024-03-25T18:21:32.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.039+0000 7fddbfb67640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb81d7490 con 0x7fddb81430c0 2024-03-25T18:21:32.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.039+0000 7fddbfb67640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb81d8360 con 0x7fddb80a98f0 2024-03-25T18:21:32.043 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.039+0000 7fddbfb67640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:32.043 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.039+0000 7fddbfb67640 10 monclient(hunting): authenticate will time out at 7986.224975s 2024-03-25T18:21:32.043 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.043+0000 7fddbe104640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fddb81430c0 0x7fddb8133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:40762/0 (socket says 172.21.15.63:40762) 2024-03-25T18:21:32.043 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.043+0000 7fddbe104640 1 -- 172.21.15.63:0/2269691784 learned_addr learned my addr 172.21.15.63:0/2269691784 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:21:32.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.044+0000 7fddae7fc640 10 client.?.objecter ms_handle_connect 0x7fddb81430c0 2024-03-25T18:21:32.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.044+0000 7fddae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:32.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.049+0000 7fddae7fc640 10 client.?.objecter ms_handle_connect 0x7fddb813f710 2024-03-25T18:21:32.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.049+0000 7fddae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:32.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.053+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159434555 0 0) 0x7fddb81d7490 con 0x7fddb81430c0 2024-03-25T18:21:32.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.053+0000 7fddae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.053+0000 7fddae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.053+0000 7fddae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.053+0000 7fddae7fc640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:21:32.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.053+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd9c004eb0 con 0x7fddb81430c0 2024-03-25T18:21:32.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.054+0000 7fddae7fc640 10 client.?.objecter ms_handle_connect 0x7fddb80a98f0 2024-03-25T18:21:32.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.054+0000 7fddae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:32.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.060+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1600098929 0 0) 0x7fdd9c004eb0 con 0x7fddb81430c0 2024-03-25T18:21:32.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.060+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddb81d7490 con 0x7fddb81430c0 2024-03-25T18:21:32.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.060+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fddb0001c90 con 0x7fddb81430c0 2024-03-25T18:21:32.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.060+0000 7fddae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:32.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.060+0000 7fddae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:32.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.060+0000 7fddae7fc640 10 monclient(hunting): dump: 2024-03-25T18:21:32.061 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:32.061 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:32.061 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:32.061 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:32.061 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:32.061 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:32.061 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:32.061 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:32.061 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:32.061 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:32.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.060+0000 7fddae7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:32.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.061+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64601434 0 0) 0x7fddb80aba00 con 0x7fddb813f710 2024-03-25T18:21:32.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.061+0000 7fddae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.061+0000 7fddae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.061+0000 7fddae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.061+0000 7fddae7fc640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:21:32.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.061+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd9c004eb0 con 0x7fddb813f710 2024-03-25T18:21:32.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.061+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036395415 0 0) 0x7fddb81d8360 con 0x7fddb80a98f0 2024-03-25T18:21:32.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.061+0000 7fddae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.061+0000 7fddae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.061+0000 7fddae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.061+0000 7fddae7fc640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:21:32.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.061+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb80aba00 con 0x7fddb80a98f0 2024-03-25T18:21:32.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3946352744 0 0) 0x7fddb81d7490 con 0x7fddb81430c0 2024-03-25T18:21:32.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 >> v1:172.21.15.63:6790/0 conn(0x7fddb80a98f0 legacy=0x7fddb81386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:32.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 >> v1:172.21.15.63:6789/0 conn(0x7fddb813f710 legacy=0x7fddb81d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:32.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddae7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:32.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddae7fc640 1 monclient: found mon.b 2024-03-25T18:21:32.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:32.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddae7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:32.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb80abc30 con 0x7fddb81430c0 2024-03-25T18:21:32.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddae7fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:32.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddae7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:32.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:32.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddbfb67640 5 monclient: authenticate success, global_id 4642 2024-03-25T18:21:32.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddbfb67640 10 monclient: _renew_subs 2024-03-25T18:21:32.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddbfb67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:32.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddbfb67640 1 -- 172.21.15.63:0/2269691784 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb8137940 con 0x7fddb81430c0 2024-03-25T18:21:32.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddbfb67640 10 client.4642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:32.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddbfb67640 10 monclient: _renew_subs 2024-03-25T18:21:32.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddbfb67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:32.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddbfb67640 1 -- 172.21.15.63:0/2269691784 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb8135220 con 0x7fddb81430c0 2024-03-25T18:21:32.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.066+0000 7fddbfb67640 1 librados: init done 2024-03-25T18:21:32.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.067+0000 7fddbfb67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:32.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.067+0000 7fddbfb67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:32.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.067+0000 7fddbfb67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:32.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.067+0000 7fddbfb67640 1 -- 172.21.15.63:0/2269691784 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb80627b0 con 0x7fddb81430c0 2024-03-25T18:21:32.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.341+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddb0003970 con 0x7fddb81430c0 2024-03-25T18:21:32.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.341+0000 7fddae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:32.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.341+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fddb0003d40 con 0x7fddb81430c0 2024-03-25T18:21:32.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.341+0000 7fddae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:32.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.341+0000 7fddae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:32.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.341+0000 7fddae7fc640 10 monclient: dump: 2024-03-25T18:21:32.341 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:32.341 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:32.342 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:32.342 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:32.342 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:32.342 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:32.342 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:32.342 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:32.342 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:32.342 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:32.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.341+0000 7fddaffff640 4 set_mon_vals no callback set 2024-03-25T18:21:32.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.341+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fddb0003f20 con 0x7fddb81430c0 2024-03-25T18:21:32.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.342+0000 7fddbd903640 10 monclient: get_auth_request con 0x7fdd9c06de80 auth_method 0 2024-03-25T18:21:32.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.342+0000 7fddae7fc640 10 client.4642.objecter ms_handle_connect 0x7fdd9c06de80 2024-03-25T18:21:32.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.347+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fddb0054ba0 con 0x7fddb81430c0 2024-03-25T18:21:32.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.347+0000 7fddae7fc640 10 client.4642.objecter ms_dispatch 0x7fddb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:32.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.347+0000 7fddae7fc640 3 client.4642.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:32.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.347+0000 7fddae7fc640 3 client.4642.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:32.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.347+0000 7fddae7fc640 20 client.4642.objecter dump_active .. 0 homeless 2024-03-25T18:21:32.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.347+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fddb00506b0 con 0x7fddb81430c0 2024-03-25T18:21:32.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.347+0000 7fddae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:32.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.347+0000 7fddae7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:32.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.569+0000 7fddbfb67640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:32.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.570+0000 7fddbfb67640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:32.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.570+0000 7fddbfb67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:32.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.570+0000 7fddbfb67640 1 -- 172.21.15.63:0/2269691784 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddb813fdc0 con 0x7fddb81430c0 2024-03-25T18:21:32.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.619+0000 7fddae7fc640 1 -- 172.21.15.63:0/2269691784 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fddb00525e0 con 0x7fddb81430c0 2024-03-25T18:21:32.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.619+0000 7fddae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:32.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.619+0000 7fddae7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:32.619 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:21:32.619 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:32.622 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.621+0000 7fddbfb67640 10 librados: watch_flush enter 2024-03-25T18:21:32.622 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.621+0000 7fddbfb67640 10 librados: watch_flush exit 2024-03-25T18:21:32.622 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.621+0000 7fddbfb67640 20 client.4642.objecter shutdown clearing up homeless session... 2024-03-25T18:21:32.622 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.621+0000 7fddbfb67640 10 client.4642.objecter successfully canceled tick 2024-03-25T18:21:32.622 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.622+0000 7fddbfb67640 1 -- 172.21.15.63:0/2269691784 >> v1:172.21.15.81:6800/3640340873 conn(0x7fdd9c06de80 legacy=0x7fdd9c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:32.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.622+0000 7fddbfb67640 10 monclient: shutdown 2024-03-25T18:21:32.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.622+0000 7fddbfb67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:32.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.622+0000 7fddbfb67640 1 -- 172.21.15.63:0/2269691784 >> v1:172.21.15.81:6789/0 conn(0x7fddb81430c0 legacy=0x7fddb8133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:32.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.622+0000 7fddbfb67640 1 -- 172.21.15.63:0/2269691784 shutdown_connections 2024-03-25T18:21:32.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.622+0000 7fddbfb67640 1 -- 172.21.15.63:0/2269691784 >> 172.21.15.63:0/2269691784 conn(0x7fddb80b7ea0 msgr2=0x7fddb8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:32.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.622+0000 7fddbfb67640 1 -- 172.21.15.63:0/2269691784 shutdown_connections 2024-03-25T18:21:32.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.622+0000 7fddbfb67640 1 -- 172.21.15.63:0/2269691784 wait complete. 2024-03-25T18:21:32.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:32.622+0000 7fddbfb67640 1 librados: shutdown 2024-03-25T18:21:32.633 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:39.440 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:39.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.550+0000 7f7868055640 10 monclient: get_monmap_and_config 2024-03-25T18:21:39.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.550+0000 7f7868055640 10 monclient: build_initial_monmap 2024-03-25T18:21:39.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.550+0000 7f7868055640 10 monclient: monmap: 2024-03-25T18:21:39.551 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:21:39.551 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:39.551 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:21:39.550988+0000 2024-03-25T18:21:39.551 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:21:39.550988+0000 2024-03-25T18:21:39.551 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:39.551 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:39.551 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:21:39.551 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:21:39.552 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:21:39.552 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:39.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.551+0000 7f7868055640 1 Processor -- start 2024-03-25T18:21:39.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.551+0000 7f7868055640 1 -- start start 2024-03-25T18:21:39.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.551+0000 7f7868055640 10 monclient: init 2024-03-25T18:21:39.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.552+0000 7f7868055640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:39.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.552+0000 7f7868055640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:39.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.552+0000 7f7868055640 10 monclient(hunting): picked mon.noname-a con 0x7f78600b0070 addr v1:172.21.15.63:6789/0 2024-03-25T18:21:39.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.552+0000 7f7868055640 10 monclient(hunting): picked mon.noname-b con 0x7f786013c170 addr v1:172.21.15.63:6790/0 2024-03-25T18:21:39.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.552+0000 7f7868055640 10 monclient(hunting): picked mon.noname-c con 0x7f786006fa90 addr v1:172.21.15.81:6789/0 2024-03-25T18:21:39.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.552+0000 7f7868055640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78600b0450 con 0x7f78600b0070 2024-03-25T18:21:39.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.552+0000 7f7868055640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7860071440 con 0x7f786006fa90 2024-03-25T18:21:39.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.552+0000 7f7868055640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78600b0bf0 con 0x7f786013c170 2024-03-25T18:21:39.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.552+0000 7f7868055640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:39.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.552+0000 7f7868055640 10 monclient(hunting): authenticate will time out at 7993.738028s 2024-03-25T18:21:39.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.567+0000 7f78655f0640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f786013c170 0x7f7860141d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:56066/0 (socket says 172.21.15.63:56066) 2024-03-25T18:21:39.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.567+0000 7f78655f0640 1 -- 172.21.15.63:0/1027431196 learned_addr learned my addr 172.21.15.63:0/1027431196 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:21:39.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.583+0000 7f7864def640 1 -- 172.21.15.63:0/1027431196 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560952342 0 0) 0x7f78600b0bf0 con 0x7f786013c170 2024-03-25T18:21:39.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.583+0000 7f7864def640 10 monclient(hunting): _negotiate 2024-03-25T18:21:39.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.583+0000 7f7864def640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:39.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.583+0000 7f7864def640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:39.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.583+0000 7f7864def640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:21:39.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.583+0000 7f7864def640 1 -- 172.21.15.63:0/1027431196 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f784c003520 con 0x7f786013c170 2024-03-25T18:21:39.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.584+0000 7f7864def640 1 -- 172.21.15.63:0/1027431196 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682214786 0 0) 0x7f7860071440 con 0x7f786006fa90 2024-03-25T18:21:39.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.584+0000 7f7864def640 10 monclient(hunting): _negotiate 2024-03-25T18:21:39.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.584+0000 7f7864def640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:39.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.584+0000 7f7864def640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:39.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.584+0000 7f7864def640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:21:39.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.584+0000 7f7864def640 1 -- 172.21.15.63:0/1027431196 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78600b0bf0 con 0x7f786006fa90 2024-03-25T18:21:39.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7864def640 1 -- 172.21.15.63:0/1027431196 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1701575884 0 0) 0x7f784c003520 con 0x7f786013c170 2024-03-25T18:21:39.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7864def640 1 -- 172.21.15.63:0/1027431196 >> v1:172.21.15.81:6789/0 conn(0x7f786006fa90 legacy=0x7f786014a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:39.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7864def640 1 -- 172.21.15.63:0/1027431196 >> v1:172.21.15.63:6789/0 conn(0x7f78600b0070 legacy=0x7f786013ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:39.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7864def640 10 monclient: _finish_hunting 0 2024-03-25T18:21:39.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7864def640 1 monclient: found mon.noname-b 2024-03-25T18:21:39.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7864def640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:39.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7864def640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:21:39.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7864def640 1 -- 172.21.15.63:0/1027431196 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78600716b0 con 0x7f786013c170 2024-03-25T18:21:39.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7864def640 10 monclient: _finish_auth 0 2024-03-25T18:21:39.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7864def640 10 monclient: _check_auth_tickets 2024-03-25T18:21:39.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7864def640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:39.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7864def640 1 -- 172.21.15.63:0/1027431196 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7850001690 con 0x7f786013c170 2024-03-25T18:21:39.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7868055640 5 monclient: authenticate success, global_id 4619 2024-03-25T18:21:39.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7868055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:39.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7864def640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:39.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7864def640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:39.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7864def640 10 monclient: dump: 2024-03-25T18:21:39.593 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:39.593 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:39.593 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:39.594 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:39.594 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:39.594 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:39.594 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:39.594 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:39.594 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:39.594 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:39.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.592+0000 7f7868055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:39.595 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.595+0000 7f7864def640 1 -- 172.21.15.63:0/1027431196 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78500017f0 con 0x7f786013c170 2024-03-25T18:21:39.595 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.595+0000 7f7864def640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:39.595 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.595+0000 7f7864def640 1 -- 172.21.15.63:0/1027431196 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7850001c00 con 0x7f786013c170 2024-03-25T18:21:39.595 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.595+0000 7f7868055640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:39.595 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.595+0000 7f7868055640 4 set_mon_vals no callback set 2024-03-25T18:21:39.595 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.595+0000 7f7864def640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:39.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.595+0000 7f7864def640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:39.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.595+0000 7f7864def640 10 monclient: dump: 2024-03-25T18:21:39.596 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:39.596 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:39.596 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:39.596 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:39.596 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:39.596 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:39.596 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:39.596 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:39.596 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:39.596 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:39.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.596+0000 7f7868055640 10 monclient: shutdown 2024-03-25T18:21:39.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.596+0000 7f7868055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:39.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.596+0000 7f7868055640 1 -- 172.21.15.63:0/1027431196 >> v1:172.21.15.63:6790/0 conn(0x7f786013c170 legacy=0x7f7860141d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:39.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.596+0000 7f7868055640 1 -- 172.21.15.63:0/1027431196 shutdown_connections 2024-03-25T18:21:39.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.596+0000 7f7868055640 1 -- 172.21.15.63:0/1027431196 >> 172.21.15.63:0/1027431196 conn(0x7f78600b0ef0 msgr2=0x7f78600b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:39.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.596+0000 7f7868055640 1 -- 172.21.15.63:0/1027431196 shutdown_connections 2024-03-25T18:21:39.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.596+0000 7f7868055640 1 -- 172.21.15.63:0/1027431196 wait complete. 2024-03-25T18:21:39.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.597+0000 7f7868055640 10 monclient: build_initial_monmap 2024-03-25T18:21:39.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.597+0000 7f7868055640 10 monclient: monmap: 2024-03-25T18:21:39.598 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:21:39.598 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:39.598 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:21:39.598 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:21:39.598 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:39.598 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:21:39.598 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:21:39.598 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:21:39.599 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:21:39.599 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:39.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.597+0000 7f7868055640 1 librados: starting msgr at 2024-03-25T18:21:39.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.597+0000 7f7868055640 1 librados: starting objecter 2024-03-25T18:21:39.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.597+0000 7f7868055640 1 Processor -- start 2024-03-25T18:21:39.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.597+0000 7f7868055640 1 -- start start 2024-03-25T18:21:39.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.597+0000 7f7868055640 1 librados: setting wanted keys 2024-03-25T18:21:39.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.597+0000 7f7868055640 1 librados: calling monclient init 2024-03-25T18:21:39.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.597+0000 7f7868055640 10 monclient: init 2024-03-25T18:21:39.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.598+0000 7f7868055640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:39.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.598+0000 7f7868055640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:39.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.598+0000 7f7868055640 10 monclient(hunting): picked mon.noname-a con 0x7f786006fa90 addr v1:172.21.15.63:6789/0 2024-03-25T18:21:39.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.598+0000 7f7868055640 10 monclient(hunting): picked mon.noname-b con 0x7f78600b0070 addr v1:172.21.15.81:6789/0 2024-03-25T18:21:39.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.598+0000 7f7868055640 10 monclient(hunting): picked mon.noname-c con 0x7f786013c170 addr v1:172.21.15.63:6790/0 2024-03-25T18:21:39.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.598+0000 7f7868055640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786013ed30 con 0x7f786006fa90 2024-03-25T18:21:39.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.598+0000 7f7868055640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786013e9c0 con 0x7f78600b0070 2024-03-25T18:21:39.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.598+0000 7f7868055640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7860141490 con 0x7f786013c170 2024-03-25T18:21:39.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.598+0000 7f7868055640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:39.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.598+0000 7f7868055640 10 monclient(hunting): authenticate will time out at 7993.783510s 2024-03-25T18:21:39.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.600+0000 7f78655f0640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f78600b0070 0x7f78601d32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:42188/0 (socket says 172.21.15.63:42188) 2024-03-25T18:21:39.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.600+0000 7f78655f0640 1 -- 172.21.15.63:0/710291204 learned_addr learned my addr 172.21.15.63:0/710291204 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:21:39.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.601+0000 7f78567fc640 10 client.?.objecter ms_handle_connect 0x7f78600b0070 2024-03-25T18:21:39.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.601+0000 7f78567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:39.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.609+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383257386 0 0) 0x7f786013e9c0 con 0x7f78600b0070 2024-03-25T18:21:39.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.609+0000 7f78567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:39.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.609+0000 7f78567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:39.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.609+0000 7f78567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:39.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.610+0000 7f78567fc640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:21:39.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.610+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7844004eb0 con 0x7f78600b0070 2024-03-25T18:21:39.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.612+0000 7f78567fc640 10 client.?.objecter ms_handle_connect 0x7f786013c170 2024-03-25T18:21:39.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.612+0000 7f78567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:39.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.618+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 864522871 0 0) 0x7f7844004eb0 con 0x7f78600b0070 2024-03-25T18:21:39.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.618+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f786013e9c0 con 0x7f78600b0070 2024-03-25T18:21:39.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.619+0000 7f78567fc640 10 client.?.objecter ms_handle_connect 0x7f786006fa90 2024-03-25T18:21:39.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.619+0000 7f78567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:39.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.619+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7850001d40 con 0x7f78600b0070 2024-03-25T18:21:39.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.619+0000 7f78567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:39.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.619+0000 7f78567fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:39.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.619+0000 7f78567fc640 10 monclient(hunting): dump: 2024-03-25T18:21:39.620 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:39.620 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:39.620 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:39.620 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:39.620 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:39.620 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:39.620 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:39.620 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:39.620 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:39.620 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:39.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.619+0000 7f78567fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:39.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.625+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967749862 0 0) 0x7f786013ed30 con 0x7f786006fa90 2024-03-25T18:21:39.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.625+0000 7f78567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:39.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.625+0000 7f78567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:39.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.625+0000 7f78567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:39.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.625+0000 7f78567fc640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:21:39.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.625+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7844004eb0 con 0x7f786006fa90 2024-03-25T18:21:39.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3471132876 0 0) 0x7f786013e9c0 con 0x7f78600b0070 2024-03-25T18:21:39.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 >> v1:172.21.15.63:6790/0 conn(0x7f786013c170 legacy=0x7f78601ddd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:39.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 >> v1:172.21.15.63:6789/0 conn(0x7f786006fa90 legacy=0x7f786013e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:39.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f78567fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:39.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f78567fc640 1 monclient: found mon.b 2024-03-25T18:21:39.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f78567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:39.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f78567fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:39.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786013dd10 con 0x7f78600b0070 2024-03-25T18:21:39.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f78567fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:39.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f78567fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:39.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f78567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:39.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f7868055640 5 monclient: authenticate success, global_id 4648 2024-03-25T18:21:39.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f7868055640 10 monclient: _renew_subs 2024-03-25T18:21:39.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f7868055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:39.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f7868055640 1 -- 172.21.15.63:0/710291204 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f786013dfe0 con 0x7f78600b0070 2024-03-25T18:21:39.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f7868055640 10 client.4648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:39.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f7868055640 10 monclient: _renew_subs 2024-03-25T18:21:39.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f7868055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:39.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f7868055640 1 -- 172.21.15.63:0/710291204 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78601d3bb0 con 0x7f78600b0070 2024-03-25T18:21:39.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.626+0000 7f7868055640 1 librados: init done 2024-03-25T18:21:39.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.627+0000 7f7868055640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:39.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.627+0000 7f7868055640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:39.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.627+0000 7f7868055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:39.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.627+0000 7f7868055640 1 -- 172.21.15.63:0/710291204 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f786013eee0 con 0x7f78600b0070 2024-03-25T18:21:39.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.648+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7850004a90 con 0x7f78600b0070 2024-03-25T18:21:39.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.648+0000 7f78567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:39.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.648+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f7850004c00 con 0x7f78600b0070 2024-03-25T18:21:39.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.648+0000 7f78567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:39.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.648+0000 7f78567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:39.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.648+0000 7f78567fc640 10 monclient: dump: 2024-03-25T18:21:39.650 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:39.650 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:39.650 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:39.650 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:39.650 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:39.650 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:39.650 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:39.650 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:39.650 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:39.650 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:39.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.648+0000 7f7857fff640 4 set_mon_vals no callback set 2024-03-25T18:21:39.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.649+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f785001abd0 con 0x7f78600b0070 2024-03-25T18:21:39.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.650+0000 7f7865df1640 10 monclient: get_auth_request con 0x7f784406dd40 auth_method 0 2024-03-25T18:21:39.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.650+0000 7f78567fc640 10 client.4648.objecter ms_handle_connect 0x7f784406dd40 2024-03-25T18:21:39.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.652+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f78500548a0 con 0x7f78600b0070 2024-03-25T18:21:39.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.652+0000 7f78567fc640 10 client.4648.objecter ms_dispatch 0x7f7860000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:39.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.652+0000 7f78567fc640 3 client.4648.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:39.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.652+0000 7f78567fc640 3 client.4648.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:39.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.652+0000 7f78567fc640 20 client.4648.objecter dump_active .. 0 homeless 2024-03-25T18:21:39.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.910+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f78500501e0 con 0x7f78600b0070 2024-03-25T18:21:39.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.910+0000 7f78567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:39.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:39.910+0000 7f78567fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:40.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.174+0000 7f7868055640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:40.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.174+0000 7f7868055640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:40.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.174+0000 7f7868055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:40.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.174+0000 7f7868055640 1 -- 172.21.15.63:0/710291204 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7860141620 con 0x7f78600b0070 2024-03-25T18:21:40.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.204+0000 7f78567fc640 1 -- 172.21.15.63:0/710291204 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f78500522e0 con 0x7f78600b0070 2024-03-25T18:21:40.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.204+0000 7f78567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:40.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.204+0000 7f78567fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:40.205 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:21:40.205 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:40.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.206+0000 7f7868055640 10 librados: watch_flush enter 2024-03-25T18:21:40.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.206+0000 7f7868055640 10 librados: watch_flush exit 2024-03-25T18:21:40.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.206+0000 7f7868055640 20 client.4648.objecter shutdown clearing up homeless session... 2024-03-25T18:21:40.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.206+0000 7f7868055640 10 client.4648.objecter successfully canceled tick 2024-03-25T18:21:40.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.206+0000 7f7868055640 1 -- 172.21.15.63:0/710291204 >> v1:172.21.15.81:6800/3640340873 conn(0x7f784406dd40 legacy=0x7f7844070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:40.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.206+0000 7f7868055640 10 monclient: shutdown 2024-03-25T18:21:40.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.206+0000 7f7868055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:40.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.206+0000 7f7868055640 1 -- 172.21.15.63:0/710291204 >> v1:172.21.15.81:6789/0 conn(0x7f78600b0070 legacy=0x7f78601d32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:40.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.207+0000 7f7868055640 1 -- 172.21.15.63:0/710291204 shutdown_connections 2024-03-25T18:21:40.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.207+0000 7f7868055640 1 -- 172.21.15.63:0/710291204 >> 172.21.15.63:0/710291204 conn(0x7f78600b0ef0 msgr2=0x7f78600615c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:40.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.207+0000 7f7868055640 1 -- 172.21.15.63:0/710291204 shutdown_connections 2024-03-25T18:21:40.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.207+0000 7f7868055640 1 -- 172.21.15.63:0/710291204 wait complete. 2024-03-25T18:21:40.209 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:40.207+0000 7f7868055640 1 librados: shutdown 2024-03-25T18:21:40.219 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:44.843 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:26.535 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:44.850 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:26.542 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:44.878 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:09:26.539 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:44.880 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:26.572 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:44.891 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:09:26.552 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:44.904 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:26.597 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:44.912 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:26.604 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:44.918 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:09:26.578 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:44.945 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:26.637 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:47.025 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:47.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.140+0000 7fd725ed6640 10 monclient: get_monmap_and_config 2024-03-25T18:21:47.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.140+0000 7fd725ed6640 10 monclient: build_initial_monmap 2024-03-25T18:21:47.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.140+0000 7fd725ed6640 10 monclient: monmap: 2024-03-25T18:21:47.141 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:21:47.142 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:47.142 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:21:47.141589+0000 2024-03-25T18:21:47.142 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:21:47.141589+0000 2024-03-25T18:21:47.142 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:47.142 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:47.142 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:21:47.142 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:21:47.142 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:21:47.142 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:47.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.141+0000 7fd725ed6640 1 Processor -- start 2024-03-25T18:21:47.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.141+0000 7fd725ed6640 1 -- start start 2024-03-25T18:21:47.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.141+0000 7fd725ed6640 10 monclient: init 2024-03-25T18:21:47.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.142+0000 7fd725ed6640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:47.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.142+0000 7fd725ed6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:47.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.142+0000 7fd725ed6640 10 monclient(hunting): picked mon.noname-c con 0x7fd7200a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:21:47.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.142+0000 7fd725ed6640 10 monclient(hunting): picked mon.noname-b con 0x7fd72013f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:21:47.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.142+0000 7fd725ed6640 10 monclient(hunting): picked mon.noname-a con 0x7fd7201430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:21:47.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.142+0000 7fd725ed6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd720142520 con 0x7fd7201430c0 2024-03-25T18:21:47.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.142+0000 7fd725ed6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7200b28a0 con 0x7fd7200a98f0 2024-03-25T18:21:47.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.142+0000 7fd725ed6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7200b2a30 con 0x7fd72013f710 2024-03-25T18:21:47.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.142+0000 7fd725ed6640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:47.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.142+0000 7fd725ed6640 10 monclient(hunting): authenticate will time out at 8001.328452s 2024-03-25T18:21:47.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.155+0000 7fd71f7fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fd7200a98f0 0x7fd7200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:46810/0 (socket says 172.21.15.63:46810) 2024-03-25T18:21:47.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.155+0000 7fd71f7fe640 1 -- 172.21.15.63:0/2534446465 learned_addr learned my addr 172.21.15.63:0/2534446465 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:21:47.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.162+0000 7fd71e7fc640 1 -- 172.21.15.63:0/2534446465 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483229291 0 0) 0x7fd7200b28a0 con 0x7fd7200a98f0 2024-03-25T18:21:47.164 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.162+0000 7fd71e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:47.164 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.162+0000 7fd71e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:47.164 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.162+0000 7fd71e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:47.164 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.162+0000 7fd71e7fc640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:21:47.164 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.162+0000 7fd71e7fc640 1 -- 172.21.15.63:0/2534446465 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd708003520 con 0x7fd7200a98f0 2024-03-25T18:21:47.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.164+0000 7fd71e7fc640 1 -- 172.21.15.63:0/2534446465 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440604192 0 0) 0x7fd720142520 con 0x7fd7201430c0 2024-03-25T18:21:47.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.164+0000 7fd71e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:47.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.164+0000 7fd71e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:47.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.164+0000 7fd71e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:47.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.164+0000 7fd71e7fc640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:21:47.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.164+0000 7fd71e7fc640 1 -- 172.21.15.63:0/2534446465 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7200b28a0 con 0x7fd7201430c0 2024-03-25T18:21:47.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd71e7fc640 1 -- 172.21.15.63:0/2534446465 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 176190698 0 0) 0x7fd7200b28a0 con 0x7fd7201430c0 2024-03-25T18:21:47.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd71e7fc640 1 -- 172.21.15.63:0/2534446465 >> v1:172.21.15.63:6790/0 conn(0x7fd72013f710 legacy=0x7fd7201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:47.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd71e7fc640 1 -- 172.21.15.63:0/2534446465 >> v1:172.21.15.81:6789/0 conn(0x7fd7200a98f0 legacy=0x7fd7200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:47.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd71e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:47.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd71e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:21:47.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd71e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:47.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd71e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:21:47.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd71e7fc640 1 -- 172.21.15.63:0/2534446465 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7200b44c0 con 0x7fd7201430c0 2024-03-25T18:21:47.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd71e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:47.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd71e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:47.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd71e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:47.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd71e7fc640 1 -- 172.21.15.63:0/2534446465 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fd7180019f0 con 0x7fd7201430c0 2024-03-25T18:21:47.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd725ed6640 5 monclient: authenticate success, global_id 4614 2024-03-25T18:21:47.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd725ed6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:47.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd71e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:47.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd71e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:47.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd71e7fc640 10 monclient: dump: 2024-03-25T18:21:47.173 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:47.173 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:47.173 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:47.173 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:47.173 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:47.173 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:47.173 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:47.173 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:47.173 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:47.174 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:47.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.170+0000 7fd725ed6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:47.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.194+0000 7fd71e7fc640 1 -- 172.21.15.63:0/2534446465 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd718001b50 con 0x7fd7201430c0 2024-03-25T18:21:47.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.194+0000 7fd71e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:47.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.194+0000 7fd71e7fc640 1 -- 172.21.15.63:0/2534446465 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fd718001f20 con 0x7fd7201430c0 2024-03-25T18:21:47.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.194+0000 7fd725ed6640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:47.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.194+0000 7fd725ed6640 4 set_mon_vals no callback set 2024-03-25T18:21:47.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.194+0000 7fd71e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:47.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.194+0000 7fd71e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:47.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.194+0000 7fd71e7fc640 10 monclient: dump: 2024-03-25T18:21:47.195 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:47.195 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:47.195 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:47.195 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:47.196 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:47.196 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:47.196 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:47.196 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:47.196 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:47.196 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:47.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.194+0000 7fd725ed6640 10 monclient: shutdown 2024-03-25T18:21:47.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.194+0000 7fd725ed6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:47.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.194+0000 7fd725ed6640 1 -- 172.21.15.63:0/2534446465 >> v1:172.21.15.63:6789/0 conn(0x7fd7201430c0 legacy=0x7fd72014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:47.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.195+0000 7fd725ed6640 1 -- 172.21.15.63:0/2534446465 shutdown_connections 2024-03-25T18:21:47.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.195+0000 7fd725ed6640 1 -- 172.21.15.63:0/2534446465 >> 172.21.15.63:0/2534446465 conn(0x7fd7200b7ea0 msgr2=0x7fd7200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:47.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.195+0000 7fd725ed6640 1 -- 172.21.15.63:0/2534446465 shutdown_connections 2024-03-25T18:21:47.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.195+0000 7fd725ed6640 1 -- 172.21.15.63:0/2534446465 wait complete. 2024-03-25T18:21:47.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.195+0000 7fd725ed6640 10 monclient: build_initial_monmap 2024-03-25T18:21:47.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.195+0000 7fd725ed6640 10 monclient: monmap: 2024-03-25T18:21:47.198 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:21:47.198 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:47.198 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:21:47.198 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:21:47.198 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:47.198 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:21:47.198 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:21:47.198 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:21:47.198 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:21:47.198 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:47.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.196+0000 7fd725ed6640 1 librados: starting msgr at 2024-03-25T18:21:47.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.196+0000 7fd725ed6640 1 librados: starting objecter 2024-03-25T18:21:47.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.196+0000 7fd725ed6640 1 Processor -- start 2024-03-25T18:21:47.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.196+0000 7fd725ed6640 1 -- start start 2024-03-25T18:21:47.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.196+0000 7fd725ed6640 1 librados: setting wanted keys 2024-03-25T18:21:47.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.196+0000 7fd725ed6640 1 librados: calling monclient init 2024-03-25T18:21:47.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.196+0000 7fd725ed6640 10 monclient: init 2024-03-25T18:21:47.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.197+0000 7fd725ed6640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:47.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.197+0000 7fd725ed6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:47.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.197+0000 7fd725ed6640 10 monclient(hunting): picked mon.noname-b con 0x7fd7200a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:21:47.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.197+0000 7fd725ed6640 10 monclient(hunting): picked mon.noname-a con 0x7fd72013f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:21:47.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.197+0000 7fd725ed6640 10 monclient(hunting): picked mon.noname-c con 0x7fd7201430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:21:47.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.197+0000 7fd725ed6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd720139950 con 0x7fd72013f710 2024-03-25T18:21:47.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.197+0000 7fd725ed6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7201d41e0 con 0x7fd7200a98f0 2024-03-25T18:21:47.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.197+0000 7fd725ed6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7201d4370 con 0x7fd7201430c0 2024-03-25T18:21:47.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.197+0000 7fd725ed6640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:47.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.197+0000 7fd725ed6640 10 monclient(hunting): authenticate will time out at 8001.383803s 2024-03-25T18:21:47.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.209+0000 7fd71ffff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fd7201430c0 0x7fd720138500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:60816/0 (socket says 172.21.15.63:60816) 2024-03-25T18:21:47.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.210+0000 7fd71ffff640 1 -- 172.21.15.63:0/1037558337 learned_addr learned my addr 172.21.15.63:0/1037558337 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:21:47.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.211+0000 7fd707fff640 10 client.?.objecter ms_handle_connect 0x7fd7201430c0 2024-03-25T18:21:47.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.211+0000 7fd707fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:47.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.216+0000 7fd707fff640 10 client.?.objecter ms_handle_connect 0x7fd7200a98f0 2024-03-25T18:21:47.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.216+0000 7fd707fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:47.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.217+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000796309 0 0) 0x7fd7201d4370 con 0x7fd7201430c0 2024-03-25T18:21:47.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.217+0000 7fd707fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:47.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.217+0000 7fd707fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:47.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.217+0000 7fd707fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:47.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.217+0000 7fd707fff640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:21:47.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.217+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd70c004eb0 con 0x7fd7201430c0 2024-03-25T18:21:47.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.223+0000 7fd707fff640 10 client.?.objecter ms_handle_connect 0x7fd72013f710 2024-03-25T18:21:47.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.223+0000 7fd707fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:47.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.231+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 630456478 0 0) 0x7fd70c004eb0 con 0x7fd7201430c0 2024-03-25T18:21:47.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.231+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7201d4370 con 0x7fd7201430c0 2024-03-25T18:21:47.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.231+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fd718001f60 con 0x7fd7201430c0 2024-03-25T18:21:47.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.231+0000 7fd707fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:47.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.231+0000 7fd707fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:47.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.231+0000 7fd707fff640 10 monclient(hunting): dump: 2024-03-25T18:21:47.233 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:47.233 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:47.233 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:47.233 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:47.233 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:47.233 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:47.233 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:47.234 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:47.234 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:47.234 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:47.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.231+0000 7fd707fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:47.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.233+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038479836 0 0) 0x7fd720139950 con 0x7fd72013f710 2024-03-25T18:21:47.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.233+0000 7fd707fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:47.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.233+0000 7fd707fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:47.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.233+0000 7fd707fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:47.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.233+0000 7fd707fff640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:21:47.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.233+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd70c004eb0 con 0x7fd72013f710 2024-03-25T18:21:47.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.238+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187219411 0 0) 0x7fd7201d41e0 con 0x7fd7200a98f0 2024-03-25T18:21:47.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.238+0000 7fd707fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:47.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.238+0000 7fd707fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:47.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.238+0000 7fd707fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:47.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.238+0000 7fd707fff640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:21:47.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.238+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd720139950 con 0x7fd7200a98f0 2024-03-25T18:21:47.241 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.239+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1396921116 0 0) 0x7fd7201d4370 con 0x7fd7201430c0 2024-03-25T18:21:47.241 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 >> v1:172.21.15.81:6789/0 conn(0x7fd7200a98f0 legacy=0x7fd7201db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:47.241 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 >> v1:172.21.15.63:6789/0 conn(0x7fd72013f710 legacy=0x7fd7201d4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:47.241 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd707fff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:47.241 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd707fff640 1 monclient: found mon.c 2024-03-25T18:21:47.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd707fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:47.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd707fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:21:47.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7201db860 con 0x7fd7201430c0 2024-03-25T18:21:47.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd707fff640 10 monclient: _finish_auth 0 2024-03-25T18:21:47.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd707fff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:47.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd707fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:47.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd725ed6640 5 monclient: authenticate success, global_id 4628 2024-03-25T18:21:47.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd725ed6640 10 monclient: _renew_subs 2024-03-25T18:21:47.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd725ed6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:21:47.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd725ed6640 1 -- 172.21.15.63:0/1037558337 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7201d4f50 con 0x7fd7201430c0 2024-03-25T18:21:47.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd725ed6640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:47.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd725ed6640 10 monclient: _renew_subs 2024-03-25T18:21:47.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd725ed6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:21:47.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd725ed6640 1 -- 172.21.15.63:0/1037558337 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd720138df0 con 0x7fd7201430c0 2024-03-25T18:21:47.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.240+0000 7fd725ed6640 1 librados: init done 2024-03-25T18:21:47.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.241+0000 7fd725ed6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:47.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.241+0000 7fd725ed6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:47.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.241+0000 7fd725ed6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:21:47.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.241+0000 7fd725ed6640 1 -- 172.21.15.63:0/1037558337 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7201dafa0 con 0x7fd7201430c0 2024-03-25T18:21:47.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.534+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd718003c20 con 0x7fd7201430c0 2024-03-25T18:21:47.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.534+0000 7fd707fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:47.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.534+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fd718003fb0 con 0x7fd7201430c0 2024-03-25T18:21:47.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.534+0000 7fd707fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:47.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.534+0000 7fd707fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:47.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.534+0000 7fd707fff640 10 monclient: dump: 2024-03-25T18:21:47.536 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:47.536 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:47.536 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:47.536 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:47.536 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:47.536 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:47.536 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:47.537 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:47.537 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:47.537 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:47.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.534+0000 7fd71dffb640 4 set_mon_vals no callback set 2024-03-25T18:21:47.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.535+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fd7180041d0 con 0x7fd7201430c0 2024-03-25T18:21:47.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.536+0000 7fd71f7fe640 10 monclient: get_auth_request con 0x7fd70c06de10 auth_method 0 2024-03-25T18:21:47.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.536+0000 7fd707fff640 10 client.4628.objecter ms_handle_connect 0x7fd70c06de10 2024-03-25T18:21:47.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.544+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fd718054c00 con 0x7fd7201430c0 2024-03-25T18:21:47.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.544+0000 7fd707fff640 10 client.4628.objecter ms_dispatch 0x7fd720000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:47.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.544+0000 7fd707fff640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:47.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.544+0000 7fd707fff640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:47.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.544+0000 7fd707fff640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-25T18:21:47.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.544+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd718050710 con 0x7fd7201430c0 2024-03-25T18:21:47.546 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.544+0000 7fd707fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:47.546 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.544+0000 7fd707fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:47.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.780+0000 7fd725ed6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:47.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.780+0000 7fd725ed6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:47.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.780+0000 7fd725ed6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:21:47.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.780+0000 7fd725ed6640 1 -- 172.21.15.63:0/1037558337 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7200abd10 con 0x7fd7201430c0 2024-03-25T18:21:47.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.798+0000 7fd707fff640 1 -- 172.21.15.63:0/1037558337 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fd718052640 con 0x7fd7201430c0 2024-03-25T18:21:47.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.798+0000 7fd707fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:47.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.798+0000 7fd707fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:47.800 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:21:47.800 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:47.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.800+0000 7fd725ed6640 10 librados: watch_flush enter 2024-03-25T18:21:47.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.800+0000 7fd725ed6640 10 librados: watch_flush exit 2024-03-25T18:21:47.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.800+0000 7fd725ed6640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-25T18:21:47.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.800+0000 7fd725ed6640 10 client.4628.objecter successfully canceled tick 2024-03-25T18:21:47.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.800+0000 7fd725ed6640 1 -- 172.21.15.63:0/1037558337 >> v1:172.21.15.81:6800/3640340873 conn(0x7fd70c06de10 legacy=0x7fd70c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:47.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.800+0000 7fd725ed6640 10 monclient: shutdown 2024-03-25T18:21:47.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.800+0000 7fd725ed6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:47.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.800+0000 7fd725ed6640 1 -- 172.21.15.63:0/1037558337 >> v1:172.21.15.63:6790/0 conn(0x7fd7201430c0 legacy=0x7fd720138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:47.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.800+0000 7fd725ed6640 1 -- 172.21.15.63:0/1037558337 shutdown_connections 2024-03-25T18:21:47.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.800+0000 7fd725ed6640 1 -- 172.21.15.63:0/1037558337 >> 172.21.15.63:0/1037558337 conn(0x7fd7200b7ea0 msgr2=0x7fd720074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:47.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.800+0000 7fd725ed6640 1 -- 172.21.15.63:0/1037558337 shutdown_connections 2024-03-25T18:21:47.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.800+0000 7fd725ed6640 1 -- 172.21.15.63:0/1037558337 wait complete. 2024-03-25T18:21:47.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:47.800+0000 7fd725ed6640 1 librados: shutdown 2024-03-25T18:21:47.813 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:54.620 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:54.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.733+0000 7fc69e927640 10 monclient: get_monmap_and_config 2024-03-25T18:21:54.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.733+0000 7fc69e927640 10 monclient: build_initial_monmap 2024-03-25T18:21:54.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.733+0000 7fc69e927640 10 monclient: monmap: 2024-03-25T18:21:54.735 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:21:54.735 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:54.735 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:21:54.735083+0000 2024-03-25T18:21:54.735 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:21:54.735083+0000 2024-03-25T18:21:54.736 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:54.736 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:54.736 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:21:54.736 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:21:54.736 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:21:54.736 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:54.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.734+0000 7fc69e927640 1 Processor -- start 2024-03-25T18:21:54.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.735+0000 7fc69e927640 1 -- start start 2024-03-25T18:21:54.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.735+0000 7fc69e927640 10 monclient: init 2024-03-25T18:21:54.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.735+0000 7fc69e927640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:54.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.735+0000 7fc69e927640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:54.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.735+0000 7fc69e927640 10 monclient(hunting): picked mon.noname-a con 0x7fc6980a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:21:54.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.735+0000 7fc69e927640 10 monclient(hunting): picked mon.noname-b con 0x7fc69813f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:21:54.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.735+0000 7fc69e927640 10 monclient(hunting): picked mon.noname-c con 0x7fc6981430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:21:54.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.735+0000 7fc69e927640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc698142520 con 0x7fc6980a98f0 2024-03-25T18:21:54.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.735+0000 7fc69e927640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6980b28a0 con 0x7fc6981430c0 2024-03-25T18:21:54.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.735+0000 7fc69e927640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6980b2a30 con 0x7fc69813f710 2024-03-25T18:21:54.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.735+0000 7fc69e927640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:54.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.735+0000 7fc69e927640 10 monclient(hunting): authenticate will time out at 8008.922247s 2024-03-25T18:21:54.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.749+0000 7fc69cec4640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc6981430c0 0x7fc69814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:46520/0 (socket says 172.21.15.63:46520) 2024-03-25T18:21:54.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.749+0000 7fc69cec4640 1 -- 172.21.15.63:0/3976149112 learned_addr learned my addr 172.21.15.63:0/3976149112 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:21:54.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.757+0000 7fc696ffd640 1 -- 172.21.15.63:0/3976149112 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318290676 0 0) 0x7fc6980b28a0 con 0x7fc6981430c0 2024-03-25T18:21:54.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.757+0000 7fc696ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:54.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.757+0000 7fc696ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:54.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.757+0000 7fc696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:54.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.757+0000 7fc696ffd640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:21:54.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.757+0000 7fc696ffd640 1 -- 172.21.15.63:0/3976149112 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc680003520 con 0x7fc6981430c0 2024-03-25T18:21:54.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.769+0000 7fc696ffd640 1 -- 172.21.15.63:0/3976149112 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266797873 0 0) 0x7fc6980b2a30 con 0x7fc69813f710 2024-03-25T18:21:54.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.769+0000 7fc696ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:54.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.769+0000 7fc696ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:54.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.769+0000 7fc696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:54.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.769+0000 7fc696ffd640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:21:54.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.769+0000 7fc696ffd640 1 -- 172.21.15.63:0/3976149112 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6980b28a0 con 0x7fc69813f710 2024-03-25T18:21:54.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc696ffd640 1 -- 172.21.15.63:0/3976149112 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3909788373 0 0) 0x7fc680003520 con 0x7fc6981430c0 2024-03-25T18:21:54.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc696ffd640 1 -- 172.21.15.63:0/3976149112 >> v1:172.21.15.63:6790/0 conn(0x7fc69813f710 legacy=0x7fc6981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:54.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc696ffd640 1 -- 172.21.15.63:0/3976149112 >> v1:172.21.15.63:6789/0 conn(0x7fc6980a98f0 legacy=0x7fc6980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:54.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc696ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:54.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc696ffd640 1 monclient: found mon.noname-c 2024-03-25T18:21:54.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc696ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:54.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc696ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:21:54.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc696ffd640 1 -- 172.21.15.63:0/3976149112 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6980b44c0 con 0x7fc6981430c0 2024-03-25T18:21:54.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc696ffd640 10 monclient: _finish_auth 0 2024-03-25T18:21:54.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc696ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:54.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc696ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:54.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc696ffd640 1 -- 172.21.15.63:0/3976149112 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc690001a10 con 0x7fc6981430c0 2024-03-25T18:21:54.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:54.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc696ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:54.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc696ffd640 10 monclient: dump: 2024-03-25T18:21:54.773 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:54.773 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:54.773 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:54.773 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:54.773 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:54.773 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:54.773 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:54.773 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:54.773 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:54.773 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:54.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc69e927640 5 monclient: authenticate success, global_id 4657 2024-03-25T18:21:54.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.770+0000 7fc69e927640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:54.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.791+0000 7fc696ffd640 1 -- 172.21.15.63:0/3976149112 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc690001b70 con 0x7fc6981430c0 2024-03-25T18:21:54.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.791+0000 7fc696ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:54.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.791+0000 7fc696ffd640 1 -- 172.21.15.63:0/3976149112 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc690001f80 con 0x7fc6981430c0 2024-03-25T18:21:54.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.791+0000 7fc69e927640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:54.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.791+0000 7fc69e927640 4 set_mon_vals no callback set 2024-03-25T18:21:54.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.791+0000 7fc696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:54.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.791+0000 7fc696ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:54.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.791+0000 7fc696ffd640 10 monclient: dump: 2024-03-25T18:21:54.793 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:54.793 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:54.793 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:54.793 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:54.793 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:54.793 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:54.793 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:54.793 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:54.794 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:54.794 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:54.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.792+0000 7fc69e927640 10 monclient: shutdown 2024-03-25T18:21:54.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.792+0000 7fc69e927640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:54.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.792+0000 7fc69e927640 1 -- 172.21.15.63:0/3976149112 >> v1:172.21.15.81:6789/0 conn(0x7fc6981430c0 legacy=0x7fc69814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:54.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.792+0000 7fc69e927640 1 -- 172.21.15.63:0/3976149112 shutdown_connections 2024-03-25T18:21:54.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.792+0000 7fc69e927640 1 -- 172.21.15.63:0/3976149112 >> 172.21.15.63:0/3976149112 conn(0x7fc6980b7ea0 msgr2=0x7fc6980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:54.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.792+0000 7fc69e927640 1 -- 172.21.15.63:0/3976149112 shutdown_connections 2024-03-25T18:21:54.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.792+0000 7fc69e927640 1 -- 172.21.15.63:0/3976149112 wait complete. 2024-03-25T18:21:54.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.793+0000 7fc69e927640 10 monclient: build_initial_monmap 2024-03-25T18:21:54.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.793+0000 7fc69e927640 10 monclient: monmap: 2024-03-25T18:21:54.796 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:21:54.796 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:54.796 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:21:54.796 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:21:54.796 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:54.796 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:21:54.796 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:21:54.796 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:21:54.796 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:21:54.796 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:54.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.794+0000 7fc69e927640 1 librados: starting msgr at 2024-03-25T18:21:54.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.794+0000 7fc69e927640 1 librados: starting objecter 2024-03-25T18:21:54.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.794+0000 7fc69e927640 1 Processor -- start 2024-03-25T18:21:54.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.794+0000 7fc69e927640 1 -- start start 2024-03-25T18:21:54.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.794+0000 7fc69e927640 1 librados: setting wanted keys 2024-03-25T18:21:54.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.794+0000 7fc69e927640 1 librados: calling monclient init 2024-03-25T18:21:54.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.794+0000 7fc69e927640 10 monclient: init 2024-03-25T18:21:54.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.795+0000 7fc69e927640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:54.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.795+0000 7fc69e927640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:54.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.795+0000 7fc69e927640 10 monclient(hunting): picked mon.noname-c con 0x7fc6980a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:21:54.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.795+0000 7fc69e927640 10 monclient(hunting): picked mon.noname-b con 0x7fc69813f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:21:54.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.795+0000 7fc69e927640 10 monclient(hunting): picked mon.noname-a con 0x7fc6981430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:21:54.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.795+0000 7fc69e927640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6981d7500 con 0x7fc6981430c0 2024-03-25T18:21:54.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.795+0000 7fc69e927640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6981394a0 con 0x7fc69813f710 2024-03-25T18:21:54.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.795+0000 7fc69e927640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc698138b50 con 0x7fc6980a98f0 2024-03-25T18:21:54.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.795+0000 7fc69e927640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:54.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.795+0000 7fc69e927640 10 monclient(hunting): authenticate will time out at 8008.981898s 2024-03-25T18:21:54.814 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.813+0000 7fc6977fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc69813f710 0x7fc6981d7c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:46548/0 (socket says 172.21.15.63:46548) 2024-03-25T18:21:54.814 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.813+0000 7fc6977fe640 1 -- 172.21.15.63:0/3294616649 learned_addr learned my addr 172.21.15.63:0/3294616649 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:21:54.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.814+0000 7fc694ff9640 10 client.?.objecter ms_handle_connect 0x7fc69813f710 2024-03-25T18:21:54.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.814+0000 7fc694ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:54.818 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.817+0000 7fc694ff9640 10 client.?.objecter ms_handle_connect 0x7fc6981430c0 2024-03-25T18:21:54.818 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.817+0000 7fc694ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:54.820 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.819+0000 7fc694ff9640 10 client.?.objecter ms_handle_connect 0x7fc6980a98f0 2024-03-25T18:21:54.820 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.819+0000 7fc694ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:54.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.824+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704390152 0 0) 0x7fc6981394a0 con 0x7fc69813f710 2024-03-25T18:21:54.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.824+0000 7fc694ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:21:54.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.824+0000 7fc694ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:54.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.824+0000 7fc694ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:54.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.824+0000 7fc694ff9640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:21:54.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.824+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc684004eb0 con 0x7fc69813f710 2024-03-25T18:21:54.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.835+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935336318 0 0) 0x7fc6981d7500 con 0x7fc6981430c0 2024-03-25T18:21:54.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.835+0000 7fc694ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:21:54.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.835+0000 7fc694ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:54.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.835+0000 7fc694ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:54.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.835+0000 7fc694ff9640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:21:54.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.835+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6981394a0 con 0x7fc6981430c0 2024-03-25T18:21:54.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.837+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068004091 0 0) 0x7fc698138b50 con 0x7fc6980a98f0 2024-03-25T18:21:54.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.837+0000 7fc694ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:21:54.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.837+0000 7fc694ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:54.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.837+0000 7fc694ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:54.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.837+0000 7fc694ff9640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:21:54.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.837+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6981d7500 con 0x7fc6980a98f0 2024-03-25T18:21:54.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.850+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3505662003 0 0) 0x7fc684004eb0 con 0x7fc69813f710 2024-03-25T18:21:54.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.850+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc698138b50 con 0x7fc69813f710 2024-03-25T18:21:54.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.850+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc68c001a10 con 0x7fc69813f710 2024-03-25T18:21:54.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.850+0000 7fc694ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:54.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.850+0000 7fc694ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:54.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.850+0000 7fc694ff9640 10 monclient(hunting): dump: 2024-03-25T18:21:54.851 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:54.851 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:54.852 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:54.852 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:54.852 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:54.852 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:54.852 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:54.852 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:54.852 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:54.852 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:54.852 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.850+0000 7fc694ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:54.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.861+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3133286529 0 0) 0x7fc6981394a0 con 0x7fc6981430c0 2024-03-25T18:21:54.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.861+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc684004eb0 con 0x7fc6981430c0 2024-03-25T18:21:54.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.861+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc690001e80 con 0x7fc6981430c0 2024-03-25T18:21:54.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.861+0000 7fc694ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:54.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.861+0000 7fc694ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:54.862 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.861+0000 7fc694ff9640 10 monclient(hunting): dump: 2024-03-25T18:21:54.862 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:54.862 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:54.863 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:54.863 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:54.863 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:54.863 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:54.863 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:54.863 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:54.863 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:54.863 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:54.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.873+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1331737785 0 0) 0x7fc6981d7500 con 0x7fc6980a98f0 2024-03-25T18:21:54.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.873+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6981394a0 con 0x7fc6980a98f0 2024-03-25T18:21:54.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.873+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc688001870 con 0x7fc6980a98f0 2024-03-25T18:21:54.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.873+0000 7fc694ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:54.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.873+0000 7fc694ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:54.874 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.873+0000 7fc694ff9640 10 monclient(hunting): dump: 2024-03-25T18:21:54.874 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:54.874 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:54.875 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:54.875 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:54.875 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:54.875 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:54.875 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:54.875 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:54.875 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:54.875 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:54.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.880+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2267511234 0 0) 0x7fc698138b50 con 0x7fc69813f710 2024-03-25T18:21:54.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.880+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 >> v1:172.21.15.63:6790/0 conn(0x7fc6980a98f0 legacy=0x7fc698138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:54.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.880+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 >> v1:172.21.15.63:6789/0 conn(0x7fc6981430c0 legacy=0x7fc698134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:54.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.880+0000 7fc694ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:21:54.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.880+0000 7fc694ff9640 1 monclient: found mon.b 2024-03-25T18:21:54.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.880+0000 7fc694ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:54.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.880+0000 7fc694ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:54.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.880+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6981d8380 con 0x7fc69813f710 2024-03-25T18:21:54.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.880+0000 7fc694ff9640 10 monclient: _finish_auth 0 2024-03-25T18:21:54.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.880+0000 7fc694ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:21:54.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.880+0000 7fc694ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:54.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.880+0000 7fc69e927640 5 monclient: authenticate success, global_id 4660 2024-03-25T18:21:54.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.880+0000 7fc69e927640 10 monclient: _renew_subs 2024-03-25T18:21:54.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.881+0000 7fc69e927640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:54.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.881+0000 7fc69e927640 1 -- 172.21.15.63:0/3294616649 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc698139710 con 0x7fc69813f710 2024-03-25T18:21:54.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.881+0000 7fc69e927640 10 client.4660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:54.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.881+0000 7fc69e927640 10 monclient: _renew_subs 2024-03-25T18:21:54.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.881+0000 7fc69e927640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:54.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.881+0000 7fc69e927640 1 -- 172.21.15.63:0/3294616649 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6981d7750 con 0x7fc69813f710 2024-03-25T18:21:54.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.881+0000 7fc69e927640 1 librados: init done 2024-03-25T18:21:54.884 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.882+0000 7fc69e927640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:54.884 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.882+0000 7fc69e927640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:54.884 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.882+0000 7fc69e927640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:54.884 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:54.882+0000 7fc69e927640 1 -- 172.21.15.63:0/3294616649 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6980627b0 con 0x7fc69813f710 2024-03-25T18:21:55.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.162+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc68c001b70 con 0x7fc69813f710 2024-03-25T18:21:55.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.162+0000 7fc694ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:55.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.162+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fc68c003a60 con 0x7fc69813f710 2024-03-25T18:21:55.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.162+0000 7fc694ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:55.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.162+0000 7fc694ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:55.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.162+0000 7fc694ff9640 10 monclient: dump: 2024-03-25T18:21:55.164 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:21:55.164 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:21:55.164 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:55.164 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:21:55.164 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:55.164 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:21:55.164 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:21:55.164 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:21:55.164 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:21:55.164 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:21:55.164 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.162+0000 7fc6967fc640 4 set_mon_vals no callback set 2024-03-25T18:21:55.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.163+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fc68c003c80 con 0x7fc69813f710 2024-03-25T18:21:55.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.163+0000 7fc697fff640 10 monclient: get_auth_request con 0x7fc68406e0b0 auth_method 0 2024-03-25T18:21:55.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.163+0000 7fc694ff9640 10 client.4660.objecter ms_handle_connect 0x7fc68406e0b0 2024-03-25T18:21:55.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.167+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fc68c054880 con 0x7fc69813f710 2024-03-25T18:21:55.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.167+0000 7fc694ff9640 10 client.4660.objecter ms_dispatch 0x7fc698000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:55.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.167+0000 7fc694ff9640 3 client.4660.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:55.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.167+0000 7fc694ff9640 3 client.4660.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:55.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.167+0000 7fc694ff9640 20 client.4660.objecter dump_active .. 0 homeless 2024-03-25T18:21:55.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.167+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc68c050390 con 0x7fc69813f710 2024-03-25T18:21:55.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.167+0000 7fc694ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:55.169 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.167+0000 7fc694ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:55.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.410+0000 7fc69e927640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:55.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.410+0000 7fc69e927640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:55.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.410+0000 7fc69e927640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:21:55.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.410+0000 7fc69e927640 1 -- 172.21.15.63:0/3294616649 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc698069f40 con 0x7fc69813f710 2024-03-25T18:21:55.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.459+0000 7fc694ff9640 1 -- 172.21.15.63:0/3294616649 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fc68c0522c0 con 0x7fc69813f710 2024-03-25T18:21:55.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.459+0000 7fc694ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:55.461 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.459+0000 7fc694ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:55.461 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:21:55.461 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:55.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.461+0000 7fc69e927640 10 librados: watch_flush enter 2024-03-25T18:21:55.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.461+0000 7fc69e927640 10 librados: watch_flush exit 2024-03-25T18:21:55.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.461+0000 7fc69e927640 20 client.4660.objecter shutdown clearing up homeless session... 2024-03-25T18:21:55.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.461+0000 7fc69e927640 10 client.4660.objecter successfully canceled tick 2024-03-25T18:21:55.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.462+0000 7fc69e927640 1 -- 172.21.15.63:0/3294616649 >> v1:172.21.15.81:6800/3640340873 conn(0x7fc68406e0b0 legacy=0x7fc684070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:55.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.462+0000 7fc69e927640 10 monclient: shutdown 2024-03-25T18:21:55.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.462+0000 7fc69e927640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:55.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.462+0000 7fc69e927640 1 -- 172.21.15.63:0/3294616649 >> v1:172.21.15.81:6789/0 conn(0x7fc69813f710 legacy=0x7fc6981d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:55.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.462+0000 7fc69e927640 1 -- 172.21.15.63:0/3294616649 shutdown_connections 2024-03-25T18:21:55.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.462+0000 7fc69e927640 1 -- 172.21.15.63:0/3294616649 >> 172.21.15.63:0/3294616649 conn(0x7fc6980b7ea0 msgr2=0x7fc69813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:55.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.462+0000 7fc69e927640 1 -- 172.21.15.63:0/3294616649 shutdown_connections 2024-03-25T18:21:55.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.462+0000 7fc69e927640 1 -- 172.21.15.63:0/3294616649 wait complete. 2024-03-25T18:21:55.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:21:55.462+0000 7fc69e927640 1 librados: shutdown 2024-03-25T18:21:55.476 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:02.281 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:02.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.397+0000 7fea51dab640 10 monclient: get_monmap_and_config 2024-03-25T18:22:02.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.397+0000 7fea51dab640 10 monclient: build_initial_monmap 2024-03-25T18:22:02.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.397+0000 7fea51dab640 10 monclient: monmap: 2024-03-25T18:22:02.399 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:22:02.399 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:02.399 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:22:02.398369+0000 2024-03-25T18:22:02.399 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:22:02.398369+0000 2024-03-25T18:22:02.399 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:02.399 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:02.399 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:22:02.399 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:22:02.399 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:22:02.399 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:02.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.398+0000 7fea51dab640 1 Processor -- start 2024-03-25T18:22:02.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.398+0000 7fea51dab640 1 -- start start 2024-03-25T18:22:02.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.398+0000 7fea51dab640 10 monclient: init 2024-03-25T18:22:02.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.398+0000 7fea51dab640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:02.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.398+0000 7fea51dab640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:02.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.398+0000 7fea51dab640 10 monclient(hunting): picked mon.noname-c con 0x7fea4c0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:22:02.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.398+0000 7fea51dab640 10 monclient(hunting): picked mon.noname-b con 0x7fea4c13f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:22:02.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.398+0000 7fea51dab640 10 monclient(hunting): picked mon.noname-a con 0x7fea4c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:22:02.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.398+0000 7fea51dab640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c142520 con 0x7fea4c1430c0 2024-03-25T18:22:02.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.398+0000 7fea51dab640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c0b28a0 con 0x7fea4c0a98f0 2024-03-25T18:22:02.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.398+0000 7fea51dab640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c0b2a30 con 0x7fea4c13f710 2024-03-25T18:22:02.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.398+0000 7fea51dab640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:02.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.398+0000 7fea51dab640 10 monclient(hunting): authenticate will time out at 8016.585214s 2024-03-25T18:22:02.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.402+0000 7fea4b7fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fea4c0a98f0 0x7fea4c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:43194/0 (socket says 172.21.15.63:43194) 2024-03-25T18:22:02.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.402+0000 7fea4b7fe640 1 -- 172.21.15.63:0/1744581559 learned_addr learned my addr 172.21.15.63:0/1744581559 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:22:02.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.413+0000 7fea4a7fc640 1 -- 172.21.15.63:0/1744581559 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130316303 0 0) 0x7fea4c0b28a0 con 0x7fea4c0a98f0 2024-03-25T18:22:02.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.413+0000 7fea4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:02.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.413+0000 7fea4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:02.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.413+0000 7fea4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:02.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.413+0000 7fea4a7fc640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:22:02.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.413+0000 7fea4a7fc640 1 -- 172.21.15.63:0/1744581559 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea34003520 con 0x7fea4c0a98f0 2024-03-25T18:22:02.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.418+0000 7fea4a7fc640 1 -- 172.21.15.63:0/1744581559 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427867855 0 0) 0x7fea4c142520 con 0x7fea4c1430c0 2024-03-25T18:22:02.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.418+0000 7fea4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:02.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.418+0000 7fea4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:02.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.418+0000 7fea4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:02.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.418+0000 7fea4a7fc640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:22:02.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.418+0000 7fea4a7fc640 1 -- 172.21.15.63:0/1744581559 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea4c0b28a0 con 0x7fea4c1430c0 2024-03-25T18:22:02.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea4a7fc640 1 -- 172.21.15.63:0/1744581559 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1314670253 0 0) 0x7fea34003520 con 0x7fea4c0a98f0 2024-03-25T18:22:02.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea4a7fc640 1 -- 172.21.15.63:0/1744581559 >> v1:172.21.15.63:6790/0 conn(0x7fea4c13f710 legacy=0x7fea4c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:02.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea4a7fc640 1 -- 172.21.15.63:0/1744581559 >> v1:172.21.15.63:6789/0 conn(0x7fea4c1430c0 legacy=0x7fea4c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:02.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea4a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:02.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea4a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:22:02.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:02.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea4a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:22:02.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea4a7fc640 1 -- 172.21.15.63:0/1744581559 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4c0b44c0 con 0x7fea4c0a98f0 2024-03-25T18:22:02.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea4a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:02.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea4a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:02.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:02.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea4a7fc640 1 -- 172.21.15.63:0/1744581559 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fea3c001690 con 0x7fea4c0a98f0 2024-03-25T18:22:02.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea51dab640 5 monclient: authenticate success, global_id 4663 2024-03-25T18:22:02.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea51dab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:02.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:02.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea4a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:02.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea4a7fc640 10 monclient: dump: 2024-03-25T18:22:02.423 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:02.423 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:02.423 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:02.423 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:02.423 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:02.423 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:02.423 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:02.423 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:02.423 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:02.423 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:02.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.420+0000 7fea51dab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:02.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.434+0000 7fea4a7fc640 1 -- 172.21.15.63:0/1744581559 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea3c0017f0 con 0x7fea4c0a98f0 2024-03-25T18:22:02.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.434+0000 7fea4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:02.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.434+0000 7fea4a7fc640 1 -- 172.21.15.63:0/1744581559 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fea3c001c00 con 0x7fea4c0a98f0 2024-03-25T18:22:02.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.434+0000 7fea51dab640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:02.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.434+0000 7fea51dab640 4 set_mon_vals no callback set 2024-03-25T18:22:02.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.434+0000 7fea4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:02.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.434+0000 7fea4a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:02.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.434+0000 7fea4a7fc640 10 monclient: dump: 2024-03-25T18:22:02.436 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:02.436 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:02.436 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:02.436 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:02.436 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:02.436 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:02.436 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:02.436 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:02.436 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:02.436 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:02.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.435+0000 7fea51dab640 10 monclient: shutdown 2024-03-25T18:22:02.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.435+0000 7fea51dab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:02.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.435+0000 7fea51dab640 1 -- 172.21.15.63:0/1744581559 >> v1:172.21.15.81:6789/0 conn(0x7fea4c0a98f0 legacy=0x7fea4c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:02.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.435+0000 7fea51dab640 1 -- 172.21.15.63:0/1744581559 shutdown_connections 2024-03-25T18:22:02.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.435+0000 7fea51dab640 1 -- 172.21.15.63:0/1744581559 >> 172.21.15.63:0/1744581559 conn(0x7fea4c0b7ea0 msgr2=0x7fea4c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:02.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.435+0000 7fea51dab640 1 -- 172.21.15.63:0/1744581559 shutdown_connections 2024-03-25T18:22:02.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.435+0000 7fea51dab640 1 -- 172.21.15.63:0/1744581559 wait complete. 2024-03-25T18:22:02.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.436+0000 7fea51dab640 10 monclient: build_initial_monmap 2024-03-25T18:22:02.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.436+0000 7fea51dab640 10 monclient: monmap: 2024-03-25T18:22:02.439 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:22:02.439 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:02.439 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:22:02.439 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:22:02.439 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:02.439 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:22:02.439 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:22:02.439 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:22:02.439 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:22:02.439 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:02.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.436+0000 7fea51dab640 1 librados: starting msgr at 2024-03-25T18:22:02.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.436+0000 7fea51dab640 1 librados: starting objecter 2024-03-25T18:22:02.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.436+0000 7fea51dab640 1 Processor -- start 2024-03-25T18:22:02.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.436+0000 7fea51dab640 1 -- start start 2024-03-25T18:22:02.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.436+0000 7fea51dab640 1 librados: setting wanted keys 2024-03-25T18:22:02.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.436+0000 7fea51dab640 1 librados: calling monclient init 2024-03-25T18:22:02.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.436+0000 7fea51dab640 10 monclient: init 2024-03-25T18:22:02.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.437+0000 7fea51dab640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:02.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.437+0000 7fea51dab640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:02.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.437+0000 7fea51dab640 10 monclient(hunting): picked mon.noname-b con 0x7fea4c0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:22:02.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.437+0000 7fea51dab640 10 monclient(hunting): picked mon.noname-a con 0x7fea4c13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:22:02.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.437+0000 7fea51dab640 10 monclient(hunting): picked mon.noname-c con 0x7fea4c1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:22:02.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.437+0000 7fea51dab640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c138250 con 0x7fea4c13f710 2024-03-25T18:22:02.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.437+0000 7fea51dab640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c137d50 con 0x7fea4c0a98f0 2024-03-25T18:22:02.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.437+0000 7fea51dab640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c137ee0 con 0x7fea4c1430c0 2024-03-25T18:22:02.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.437+0000 7fea51dab640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:02.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.437+0000 7fea51dab640 10 monclient(hunting): authenticate will time out at 8016.624179s 2024-03-25T18:22:02.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.443+0000 7fea4b7fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fea4c0a98f0 0x7fea4c137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:43204/0 (socket says 172.21.15.63:43204) 2024-03-25T18:22:02.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.443+0000 7fea4b7fe640 1 -- 172.21.15.63:0/3114015440 learned_addr learned my addr 172.21.15.63:0/3114015440 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:22:02.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.447+0000 7fea33fff640 10 client.?.objecter ms_handle_connect 0x7fea4c0a98f0 2024-03-25T18:22:02.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.447+0000 7fea33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:02.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.447+0000 7fea33fff640 10 client.?.objecter ms_handle_connect 0x7fea4c1430c0 2024-03-25T18:22:02.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.447+0000 7fea33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:02.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.453+0000 7fea33fff640 10 client.?.objecter ms_handle_connect 0x7fea4c13f710 2024-03-25T18:22:02.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.453+0000 7fea33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:02.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.456+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738266939 0 0) 0x7fea4c137d50 con 0x7fea4c0a98f0 2024-03-25T18:22:02.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.456+0000 7fea33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:02.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.456+0000 7fea33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:02.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.456+0000 7fea33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:02.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.456+0000 7fea33fff640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:22:02.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.456+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea38004eb0 con 0x7fea4c0a98f0 2024-03-25T18:22:02.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.463+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113632081 0 0) 0x7fea4c137ee0 con 0x7fea4c1430c0 2024-03-25T18:22:02.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.463+0000 7fea33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:02.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.463+0000 7fea33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:02.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.463+0000 7fea33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:02.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.463+0000 7fea33fff640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:22:02.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.463+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea4c137d50 con 0x7fea4c1430c0 2024-03-25T18:22:02.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.463+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4076546610 0 0) 0x7fea38004eb0 con 0x7fea4c0a98f0 2024-03-25T18:22:02.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.463+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea4c137ee0 con 0x7fea4c0a98f0 2024-03-25T18:22:02.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.463+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fea3c001c00 con 0x7fea4c0a98f0 2024-03-25T18:22:02.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.463+0000 7fea33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:02.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.463+0000 7fea33fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:02.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.463+0000 7fea33fff640 10 monclient(hunting): dump: 2024-03-25T18:22:02.465 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:02.465 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:02.465 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:02.465 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:02.465 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:02.466 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:02.466 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:02.466 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:02.466 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:02.466 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:02.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.463+0000 7fea33fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:02.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4178749876 0 0) 0x7fea4c137ee0 con 0x7fea4c0a98f0 2024-03-25T18:22:02.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 >> v1:172.21.15.63:6790/0 conn(0x7fea4c1430c0 legacy=0x7fea4c1d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:02.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 >> v1:172.21.15.63:6789/0 conn(0x7fea4c13f710 legacy=0x7fea4c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:02.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea33fff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:02.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea33fff640 1 monclient: found mon.b 2024-03-25T18:22:02.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea33fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4c0b7730 con 0x7fea4c0a98f0 2024-03-25T18:22:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea33fff640 10 monclient: _finish_auth 0 2024-03-25T18:22:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea33fff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea51dab640 5 monclient: authenticate success, global_id 4666 2024-03-25T18:22:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea51dab640 10 monclient: _renew_subs 2024-03-25T18:22:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea51dab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea51dab640 1 -- 172.21.15.63:0/3114015440 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4c0b7920 con 0x7fea4c0a98f0 2024-03-25T18:22:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea51dab640 10 client.4666.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea51dab640 10 monclient: _renew_subs 2024-03-25T18:22:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea51dab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea51dab640 1 -- 172.21.15.63:0/3114015440 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4c1e33d0 con 0x7fea4c0a98f0 2024-03-25T18:22:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.469+0000 7fea51dab640 1 librados: init done 2024-03-25T18:22:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.470+0000 7fea51dab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:02.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.470+0000 7fea51dab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:02.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.470+0000 7fea51dab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:02.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.470+0000 7fea51dab640 1 -- 172.21.15.63:0/3114015440 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea4c0627b0 con 0x7fea4c0a98f0 2024-03-25T18:22:02.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.738+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea3c004a90 con 0x7fea4c0a98f0 2024-03-25T18:22:02.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.738+0000 7fea33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:02.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.738+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fea3c004e20 con 0x7fea4c0a98f0 2024-03-25T18:22:02.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.738+0000 7fea33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:02.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.738+0000 7fea33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:02.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.738+0000 7fea33fff640 10 monclient: dump: 2024-03-25T18:22:02.741 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:02.741 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:02.741 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:02.741 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:02.741 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:02.741 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:02.741 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:02.741 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:02.741 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:02.741 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:02.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.738+0000 7fea49ffb640 4 set_mon_vals no callback set 2024-03-25T18:22:02.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.739+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fea3c005040 con 0x7fea4c0a98f0 2024-03-25T18:22:02.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.740+0000 7fea4affd640 10 monclient: get_auth_request con 0x7fea3806dd10 auth_method 0 2024-03-25T18:22:02.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.740+0000 7fea33fff640 10 client.4666.objecter ms_handle_connect 0x7fea3806dd10 2024-03-25T18:22:02.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.744+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7fea3c0549b0 con 0x7fea4c0a98f0 2024-03-25T18:22:02.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.744+0000 7fea33fff640 10 client.4666.objecter ms_dispatch 0x7fea4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:02.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.744+0000 7fea33fff640 3 client.4666.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:02.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.744+0000 7fea33fff640 3 client.4666.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:02.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.745+0000 7fea33fff640 20 client.4666.objecter dump_active .. 0 homeless 2024-03-25T18:22:02.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.746+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea3c0504c0 con 0x7fea4c0a98f0 2024-03-25T18:22:02.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.746+0000 7fea33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:02.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.746+0000 7fea33fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:02.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.969+0000 7fea51dab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:02.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.969+0000 7fea51dab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:02.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.969+0000 7fea51dab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:02.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:02.969+0000 7fea51dab640 1 -- 172.21.15.63:0/3114015440 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea4c13df50 con 0x7fea4c0a98f0 2024-03-25T18:22:03.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:03.003+0000 7fea33fff640 1 -- 172.21.15.63:0/3114015440 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7fea3c0523f0 con 0x7fea4c0a98f0 2024-03-25T18:22:03.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:03.003+0000 7fea33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:03.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:03.003+0000 7fea33fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:03.004 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:22:03.005 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:03.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:03.005+0000 7fea51dab640 10 librados: watch_flush enter 2024-03-25T18:22:03.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:03.005+0000 7fea51dab640 10 librados: watch_flush exit 2024-03-25T18:22:03.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:03.005+0000 7fea51dab640 20 client.4666.objecter shutdown clearing up homeless session... 2024-03-25T18:22:03.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:03.005+0000 7fea51dab640 10 client.4666.objecter successfully canceled tick 2024-03-25T18:22:03.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:03.006+0000 7fea51dab640 1 -- 172.21.15.63:0/3114015440 >> v1:172.21.15.81:6800/3640340873 conn(0x7fea3806dd10 legacy=0x7fea380701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:03.006+0000 7fea51dab640 10 monclient: shutdown 2024-03-25T18:22:03.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:03.006+0000 7fea51dab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:03.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:03.006+0000 7fea51dab640 1 -- 172.21.15.63:0/3114015440 >> v1:172.21.15.81:6789/0 conn(0x7fea4c0a98f0 legacy=0x7fea4c137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:03.006+0000 7fea51dab640 1 -- 172.21.15.63:0/3114015440 shutdown_connections 2024-03-25T18:22:03.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:03.006+0000 7fea51dab640 1 -- 172.21.15.63:0/3114015440 >> 172.21.15.63:0/3114015440 conn(0x7fea4c0b7ea0 msgr2=0x7fea4c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:03.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:03.006+0000 7fea51dab640 1 -- 172.21.15.63:0/3114015440 shutdown_connections 2024-03-25T18:22:03.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:03.006+0000 7fea51dab640 1 -- 172.21.15.63:0/3114015440 wait complete. 2024-03-25T18:22:03.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:03.006+0000 7fea51dab640 1 librados: shutdown 2024-03-25T18:22:03.018 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:04.838 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:46.530 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:04.881 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:09:46.542 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:04.892 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:09:46.552 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:04.903 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:46.596 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:04.931 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:09:46.591 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:04.932 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:46.624 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:04.973 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:46.665 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:04.976 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:46.669 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:05.003 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:46.696 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:09.825 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:09.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.940+0000 7f8d762c9640 10 monclient: get_monmap_and_config 2024-03-25T18:22:09.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.940+0000 7f8d762c9640 10 monclient: build_initial_monmap 2024-03-25T18:22:09.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.940+0000 7f8d762c9640 10 monclient: monmap: 2024-03-25T18:22:09.942 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:22:09.942 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:09.942 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:22:09.941670+0000 2024-03-25T18:22:09.942 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:22:09.941670+0000 2024-03-25T18:22:09.942 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:09.942 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:09.942 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:22:09.942 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:22:09.943 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:22:09.943 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:09.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.941+0000 7f8d762c9640 1 Processor -- start 2024-03-25T18:22:09.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.941+0000 7f8d762c9640 1 -- start start 2024-03-25T18:22:09.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.941+0000 7f8d762c9640 10 monclient: init 2024-03-25T18:22:09.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.942+0000 7f8d762c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:09.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.942+0000 7f8d762c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:09.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.942+0000 7f8d762c9640 10 monclient(hunting): picked mon.noname-a con 0x7f8d700a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:22:09.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.942+0000 7f8d762c9640 10 monclient(hunting): picked mon.noname-b con 0x7f8d7013f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:22:09.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.942+0000 7f8d762c9640 10 monclient(hunting): picked mon.noname-c con 0x7f8d701430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:22:09.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.942+0000 7f8d762c9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d70142520 con 0x7f8d700a98f0 2024-03-25T18:22:09.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.942+0000 7f8d762c9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d700b28a0 con 0x7f8d701430c0 2024-03-25T18:22:09.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.942+0000 7f8d762c9640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d700b2a30 con 0x7f8d7013f710 2024-03-25T18:22:09.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.942+0000 7f8d762c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:09.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.942+0000 7f8d762c9640 10 monclient(hunting): authenticate will time out at 8024.129044s 2024-03-25T18:22:09.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.959+0000 7f8d6ffff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f8d700a98f0 0x7f8d700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:46922/0 (socket says 172.21.15.63:46922) 2024-03-25T18:22:09.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.959+0000 7f8d6ffff640 1 -- 172.21.15.63:0/1824874052 learned_addr learned my addr 172.21.15.63:0/1824874052 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:22:09.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.966+0000 7f8d6effd640 1 -- 172.21.15.63:0/1824874052 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777689149 0 0) 0x7f8d700b28a0 con 0x7f8d701430c0 2024-03-25T18:22:09.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.966+0000 7f8d6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:09.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.966+0000 7f8d6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:09.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.966+0000 7f8d6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:09.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.966+0000 7f8d6effd640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:22:09.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.966+0000 7f8d6effd640 1 -- 172.21.15.63:0/1824874052 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d58003520 con 0x7f8d701430c0 2024-03-25T18:22:09.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.969+0000 7f8d6effd640 1 -- 172.21.15.63:0/1824874052 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3256178113 0 0) 0x7f8d58003520 con 0x7f8d701430c0 2024-03-25T18:22:09.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.969+0000 7f8d6effd640 1 -- 172.21.15.63:0/1824874052 >> v1:172.21.15.63:6790/0 conn(0x7f8d7013f710 legacy=0x7f8d701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:09.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.969+0000 7f8d6effd640 1 -- 172.21.15.63:0/1824874052 >> v1:172.21.15.63:6789/0 conn(0x7f8d700a98f0 legacy=0x7f8d700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:09.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.969+0000 7f8d6effd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:09.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.969+0000 7f8d6effd640 1 monclient: found mon.noname-c 2024-03-25T18:22:09.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.969+0000 7f8d6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:09.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.970+0000 7f8d6effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:22:09.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.970+0000 7f8d6effd640 1 -- 172.21.15.63:0/1824874052 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d700b44c0 con 0x7f8d701430c0 2024-03-25T18:22:09.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.970+0000 7f8d6effd640 10 monclient: _finish_auth 0 2024-03-25T18:22:09.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.970+0000 7f8d6effd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:09.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.970+0000 7f8d6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:09.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.970+0000 7f8d762c9640 5 monclient: authenticate success, global_id 4669 2024-03-25T18:22:09.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.970+0000 7f8d762c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:09.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.973+0000 7f8d6effd640 1 -- 172.21.15.63:0/1824874052 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8d68001a10 con 0x7f8d701430c0 2024-03-25T18:22:09.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.973+0000 7f8d6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:09.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.973+0000 7f8d6effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:09.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.973+0000 7f8d6effd640 10 monclient: dump: 2024-03-25T18:22:09.975 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:09.975 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:09.975 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:09.975 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:09.975 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:09.975 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:09.975 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:09.975 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:09.975 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:09.975 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:09.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.973+0000 7f8d762c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:09.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.975+0000 7f8d6effd640 1 -- 172.21.15.63:0/1824874052 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d68001ca0 con 0x7f8d701430c0 2024-03-25T18:22:09.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.976+0000 7f8d6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:09.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.976+0000 7f8d6effd640 1 -- 172.21.15.63:0/1824874052 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8d68001fb0 con 0x7f8d701430c0 2024-03-25T18:22:09.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.976+0000 7f8d762c9640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:09.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.976+0000 7f8d762c9640 4 set_mon_vals no callback set 2024-03-25T18:22:09.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.976+0000 7f8d6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:09.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.976+0000 7f8d6effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:09.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.976+0000 7f8d6effd640 10 monclient: dump: 2024-03-25T18:22:09.977 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:09.977 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:09.977 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:09.977 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:09.978 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:09.978 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:09.978 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:09.978 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:09.978 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:09.978 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:09.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.976+0000 7f8d762c9640 10 monclient: shutdown 2024-03-25T18:22:09.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.976+0000 7f8d762c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:09.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.976+0000 7f8d762c9640 1 -- 172.21.15.63:0/1824874052 >> v1:172.21.15.81:6789/0 conn(0x7f8d701430c0 legacy=0x7f8d7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:09.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.976+0000 7f8d762c9640 1 -- 172.21.15.63:0/1824874052 shutdown_connections 2024-03-25T18:22:09.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.976+0000 7f8d762c9640 1 -- 172.21.15.63:0/1824874052 >> 172.21.15.63:0/1824874052 conn(0x7f8d700b7ea0 msgr2=0x7f8d700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:09.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.976+0000 7f8d762c9640 1 -- 172.21.15.63:0/1824874052 shutdown_connections 2024-03-25T18:22:09.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.977+0000 7f8d762c9640 1 -- 172.21.15.63:0/1824874052 wait complete. 2024-03-25T18:22:09.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.977+0000 7f8d762c9640 10 monclient: build_initial_monmap 2024-03-25T18:22:09.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.977+0000 7f8d762c9640 10 monclient: monmap: 2024-03-25T18:22:09.980 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:22:09.980 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:09.980 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:22:09.980 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:22:09.980 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:09.980 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:22:09.980 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:22:09.980 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:22:09.981 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:22:09.981 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:09.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.978+0000 7f8d762c9640 1 librados: starting msgr at 2024-03-25T18:22:09.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.978+0000 7f8d762c9640 1 librados: starting objecter 2024-03-25T18:22:09.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.978+0000 7f8d762c9640 1 Processor -- start 2024-03-25T18:22:09.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.978+0000 7f8d762c9640 1 -- start start 2024-03-25T18:22:09.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.978+0000 7f8d762c9640 1 librados: setting wanted keys 2024-03-25T18:22:09.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.978+0000 7f8d762c9640 1 librados: calling monclient init 2024-03-25T18:22:09.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.978+0000 7f8d762c9640 10 monclient: init 2024-03-25T18:22:09.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.979+0000 7f8d762c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:09.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.979+0000 7f8d762c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:09.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.979+0000 7f8d762c9640 10 monclient(hunting): picked mon.noname-a con 0x7f8d700a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:22:09.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.979+0000 7f8d762c9640 10 monclient(hunting): picked mon.noname-c con 0x7f8d7013f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:22:09.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.979+0000 7f8d762c9640 10 monclient(hunting): picked mon.noname-b con 0x7f8d701430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:22:09.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.979+0000 7f8d762c9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d700b7120 con 0x7f8d700a98f0 2024-03-25T18:22:09.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.979+0000 7f8d762c9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7013b380 con 0x7f8d701430c0 2024-03-25T18:22:09.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.979+0000 7f8d762c9640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7013b660 con 0x7f8d7013f710 2024-03-25T18:22:09.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.979+0000 7f8d762c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:09.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.979+0000 7f8d762c9640 10 monclient(hunting): authenticate will time out at 8024.165822s 2024-03-25T18:22:09.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.985+0000 7f8d74866640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f8d701430c0 0x7f8d701e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:42420/0 (socket says 172.21.15.63:42420) 2024-03-25T18:22:09.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.985+0000 7f8d74866640 1 -- 172.21.15.63:0/3779570910 learned_addr learned my addr 172.21.15.63:0/3779570910 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:22:09.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.986+0000 7f8d6cff9640 10 client.?.objecter ms_handle_connect 0x7f8d701430c0 2024-03-25T18:22:09.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.986+0000 7f8d6cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:09.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.989+0000 7f8d6cff9640 10 client.?.objecter ms_handle_connect 0x7f8d7013f710 2024-03-25T18:22:09.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.989+0000 7f8d6cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:09.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.990+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715126082 0 0) 0x7f8d7013b380 con 0x7f8d701430c0 2024-03-25T18:22:09.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.990+0000 7f8d6cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:09.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.990+0000 7f8d6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:09.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.990+0000 7f8d6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:09.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.990+0000 7f8d6cff9640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:22:09.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.990+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d5c004eb0 con 0x7f8d701430c0 2024-03-25T18:22:09.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.998+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3067167709 0 0) 0x7f8d5c004eb0 con 0x7f8d701430c0 2024-03-25T18:22:10.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.998+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d7013b380 con 0x7f8d701430c0 2024-03-25T18:22:10.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.998+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8d680019d0 con 0x7f8d701430c0 2024-03-25T18:22:10.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.998+0000 7f8d6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:10.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.998+0000 7f8d6cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:10.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.998+0000 7f8d6cff9640 10 monclient(hunting): dump: 2024-03-25T18:22:10.000 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:10.000 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:10.000 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:10.000 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:10.000 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:10.000 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:10.000 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:10.001 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:10.001 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:10.001 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:10.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:09.998+0000 7f8d6cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:10.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.001+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220493593 0 0) 0x7f8d7013b660 con 0x7f8d7013f710 2024-03-25T18:22:10.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.001+0000 7f8d6cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:10.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.001+0000 7f8d6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:10.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.001+0000 7f8d6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:10.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.001+0000 7f8d6cff9640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:22:10.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.001+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d5c004eb0 con 0x7f8d7013f710 2024-03-25T18:22:10.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 277486090 0 0) 0x7f8d7013b380 con 0x7f8d701430c0 2024-03-25T18:22:10.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 >> v1:172.21.15.63:6790/0 conn(0x7f8d7013f710 legacy=0x7f8d70139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:10.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 >> v1:172.21.15.63:6789/0 conn(0x7f8d700a98f0 legacy=0x7f8d7013ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:10.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d6cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:22:10.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d6cff9640 1 monclient: found mon.b 2024-03-25T18:22:10.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d6cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:10.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d6cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:10.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d700b73c0 con 0x7f8d701430c0 2024-03-25T18:22:10.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d6cff9640 10 monclient: _finish_auth 0 2024-03-25T18:22:10.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d6cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:22:10.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d6cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:10.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d762c9640 5 monclient: authenticate success, global_id 4672 2024-03-25T18:22:10.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d762c9640 10 monclient: _renew_subs 2024-03-25T18:22:10.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d762c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:10.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d762c9640 1 -- 172.21.15.63:0/3779570910 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d700b7c30 con 0x7f8d701430c0 2024-03-25T18:22:10.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d762c9640 10 client.4672.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:10.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d762c9640 10 monclient: _renew_subs 2024-03-25T18:22:10.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d762c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:10.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d762c9640 1 -- 172.21.15.63:0/3779570910 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7012f350 con 0x7f8d701430c0 2024-03-25T18:22:10.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.010+0000 7f8d762c9640 1 librados: init done 2024-03-25T18:22:10.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.011+0000 7f8d762c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:10.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.011+0000 7f8d762c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:10.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.011+0000 7f8d762c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:10.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.011+0000 7f8d762c9640 1 -- 172.21.15.63:0/3779570910 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d700627b0 con 0x7f8d701430c0 2024-03-25T18:22:10.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.016+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d68003c20 con 0x7f8d701430c0 2024-03-25T18:22:10.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.016+0000 7f8d6cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:10.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.016+0000 7f8d6e7fc640 4 set_mon_vals no callback set 2024-03-25T18:22:10.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.289+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8d680015d0 con 0x7f8d701430c0 2024-03-25T18:22:10.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.289+0000 7f8d6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:10.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.289+0000 7f8d6cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:10.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.289+0000 7f8d6cff9640 10 monclient: dump: 2024-03-25T18:22:10.291 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:10.291 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:10.291 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:10.291 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:10.291 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:10.291 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:10.291 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:10.292 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:10.292 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:10.292 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:10.292 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.290+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f8d68004070 con 0x7f8d701430c0 2024-03-25T18:22:10.292 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.291+0000 7f8d6ffff640 10 monclient: get_auth_request con 0x7f8d5c06dd50 auth_method 0 2024-03-25T18:22:10.292 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.291+0000 7f8d6cff9640 10 client.4672.objecter ms_handle_connect 0x7f8d5c06dd50 2024-03-25T18:22:10.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.293+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f8d68054c40 con 0x7f8d701430c0 2024-03-25T18:22:10.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.293+0000 7f8d6cff9640 10 client.4672.objecter ms_dispatch 0x7f8d70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:10.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.293+0000 7f8d6cff9640 3 client.4672.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:10.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.293+0000 7f8d6cff9640 3 client.4672.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:10.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.293+0000 7f8d6cff9640 20 client.4672.objecter dump_active .. 0 homeless 2024-03-25T18:22:10.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.297+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d68050750 con 0x7f8d701430c0 2024-03-25T18:22:10.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.297+0000 7f8d6cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:10.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.297+0000 7f8d6cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:10.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.527+0000 7f8d762c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:10.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.527+0000 7f8d762c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:10.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.527+0000 7f8d762c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:10.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.527+0000 7f8d762c9640 1 -- 172.21.15.63:0/3779570910 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d7013f0a0 con 0x7f8d701430c0 2024-03-25T18:22:10.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.554+0000 7f8d6cff9640 1 -- 172.21.15.63:0/3779570910 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f8d68052680 con 0x7f8d701430c0 2024-03-25T18:22:10.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.554+0000 7f8d6cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:10.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.554+0000 7f8d6cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:10.556 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:22:10.556 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:10.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.557+0000 7f8d762c9640 10 librados: watch_flush enter 2024-03-25T18:22:10.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.557+0000 7f8d762c9640 10 librados: watch_flush exit 2024-03-25T18:22:10.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.557+0000 7f8d762c9640 20 client.4672.objecter shutdown clearing up homeless session... 2024-03-25T18:22:10.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.557+0000 7f8d762c9640 10 client.4672.objecter successfully canceled tick 2024-03-25T18:22:10.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.557+0000 7f8d762c9640 1 -- 172.21.15.63:0/3779570910 >> v1:172.21.15.81:6800/3640340873 conn(0x7f8d5c06dd50 legacy=0x7f8d5c070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:10.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.557+0000 7f8d762c9640 10 monclient: shutdown 2024-03-25T18:22:10.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.557+0000 7f8d762c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:10.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.557+0000 7f8d762c9640 1 -- 172.21.15.63:0/3779570910 >> v1:172.21.15.81:6789/0 conn(0x7f8d701430c0 legacy=0x7f8d701e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:10.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.557+0000 7f8d762c9640 1 -- 172.21.15.63:0/3779570910 shutdown_connections 2024-03-25T18:22:10.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.557+0000 7f8d762c9640 1 -- 172.21.15.63:0/3779570910 >> 172.21.15.63:0/3779570910 conn(0x7f8d700b7ea0 msgr2=0x7f8d70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:10.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.557+0000 7f8d762c9640 1 -- 172.21.15.63:0/3779570910 shutdown_connections 2024-03-25T18:22:10.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.557+0000 7f8d762c9640 1 -- 172.21.15.63:0/3779570910 wait complete. 2024-03-25T18:22:10.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:10.557+0000 7f8d762c9640 1 librados: shutdown 2024-03-25T18:22:10.571 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:17.376 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:17.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.489+0000 7f8ee204a640 10 monclient: get_monmap_and_config 2024-03-25T18:22:17.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.489+0000 7f8ee204a640 10 monclient: build_initial_monmap 2024-03-25T18:22:17.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.489+0000 7f8ee204a640 10 monclient: monmap: 2024-03-25T18:22:17.490 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:22:17.490 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:17.490 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:22:17.490335+0000 2024-03-25T18:22:17.490 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:22:17.490335+0000 2024-03-25T18:22:17.490 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:17.491 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:17.491 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:22:17.491 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:22:17.491 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:22:17.491 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:17.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.490+0000 7f8ee204a640 1 Processor -- start 2024-03-25T18:22:17.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.490+0000 7f8ee204a640 1 -- start start 2024-03-25T18:22:17.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.490+0000 7f8ee204a640 10 monclient: init 2024-03-25T18:22:17.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.491+0000 7f8ee204a640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:17.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.491+0000 7f8ee204a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:17.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.491+0000 7f8ee204a640 10 monclient(hunting): picked mon.noname-c con 0x7f8edc0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:22:17.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.491+0000 7f8ee204a640 10 monclient(hunting): picked mon.noname-a con 0x7f8edc13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:22:17.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.491+0000 7f8ee204a640 10 monclient(hunting): picked mon.noname-b con 0x7f8edc1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:22:17.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.491+0000 7f8ee204a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8edc142520 con 0x7f8edc13f710 2024-03-25T18:22:17.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.491+0000 7f8ee204a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8edc0b28a0 con 0x7f8edc0a98f0 2024-03-25T18:22:17.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.491+0000 7f8ee204a640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8edc0b2a30 con 0x7f8edc1430c0 2024-03-25T18:22:17.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.491+0000 7f8ee204a640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:17.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.491+0000 7f8ee204a640 10 monclient(hunting): authenticate will time out at 8031.677751s 2024-03-25T18:22:17.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.502+0000 7f8edb7fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f8edc0a98f0 0x7f8edc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:47084/0 (socket says 172.21.15.63:47084) 2024-03-25T18:22:17.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.502+0000 7f8edb7fe640 1 -- 172.21.15.63:0/4288087022 learned_addr learned my addr 172.21.15.63:0/4288087022 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:22:17.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.507+0000 7f8eda7fc640 1 -- 172.21.15.63:0/4288087022 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757406574 0 0) 0x7f8edc0b2a30 con 0x7f8edc1430c0 2024-03-25T18:22:17.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.508+0000 7f8eda7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:17.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.508+0000 7f8eda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:17.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.508+0000 7f8eda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:17.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.508+0000 7f8eda7fc640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:22:17.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.508+0000 7f8eda7fc640 1 -- 172.21.15.63:0/4288087022 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec4003520 con 0x7f8edc1430c0 2024-03-25T18:22:17.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8eda7fc640 1 -- 172.21.15.63:0/4288087022 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2938077505 0 0) 0x7f8ec4003520 con 0x7f8edc1430c0 2024-03-25T18:22:17.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8eda7fc640 1 -- 172.21.15.63:0/4288087022 >> v1:172.21.15.81:6789/0 conn(0x7f8edc0a98f0 legacy=0x7f8edc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:17.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8eda7fc640 1 -- 172.21.15.63:0/4288087022 >> v1:172.21.15.63:6789/0 conn(0x7f8edc13f710 legacy=0x7f8edc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:17.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8eda7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:17.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8eda7fc640 1 monclient: found mon.noname-b 2024-03-25T18:22:17.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8eda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:17.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8eda7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:22:17.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8eda7fc640 1 -- 172.21.15.63:0/4288087022 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8edc0b44c0 con 0x7f8edc1430c0 2024-03-25T18:22:17.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8eda7fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:17.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8eda7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:17.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8eda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:17.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8eda7fc640 1 -- 172.21.15.63:0/4288087022 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8ed4001a10 con 0x7f8edc1430c0 2024-03-25T18:22:17.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8ee204a640 5 monclient: authenticate success, global_id 4649 2024-03-25T18:22:17.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8ee204a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:17.519 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8eda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:17.519 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8eda7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:17.519 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8eda7fc640 10 monclient: dump: 2024-03-25T18:22:17.519 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:17.519 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:17.519 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:17.519 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:17.519 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:17.519 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:17.519 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:17.519 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:17.519 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:17.519 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:17.519 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.516+0000 7f8ee204a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:17.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.525+0000 7f8eda7fc640 1 -- 172.21.15.63:0/4288087022 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ed4001b70 con 0x7f8edc1430c0 2024-03-25T18:22:17.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.525+0000 7f8eda7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:17.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.525+0000 7f8eda7fc640 1 -- 172.21.15.63:0/4288087022 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8ed4001f80 con 0x7f8edc1430c0 2024-03-25T18:22:17.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.525+0000 7f8eda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:17.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.525+0000 7f8eda7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:17.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.525+0000 7f8eda7fc640 10 monclient: dump: 2024-03-25T18:22:17.527 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:17.527 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:17.527 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:17.527 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:17.527 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:17.527 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:17.527 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:17.527 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:17.527 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:17.527 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:17.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.525+0000 7f8ee204a640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:17.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.525+0000 7f8ee204a640 4 set_mon_vals no callback set 2024-03-25T18:22:17.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.526+0000 7f8ee204a640 10 monclient: shutdown 2024-03-25T18:22:17.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.526+0000 7f8ee204a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:17.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.526+0000 7f8ee204a640 1 -- 172.21.15.63:0/4288087022 >> v1:172.21.15.63:6790/0 conn(0x7f8edc1430c0 legacy=0x7f8edc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:17.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.526+0000 7f8ee204a640 1 -- 172.21.15.63:0/4288087022 shutdown_connections 2024-03-25T18:22:17.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.526+0000 7f8ee204a640 1 -- 172.21.15.63:0/4288087022 >> 172.21.15.63:0/4288087022 conn(0x7f8edc0b7ea0 msgr2=0x7f8edc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:17.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.526+0000 7f8ee204a640 1 -- 172.21.15.63:0/4288087022 shutdown_connections 2024-03-25T18:22:17.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.526+0000 7f8ee204a640 1 -- 172.21.15.63:0/4288087022 wait complete. 2024-03-25T18:22:17.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.527+0000 7f8ee204a640 10 monclient: build_initial_monmap 2024-03-25T18:22:17.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.527+0000 7f8ee204a640 10 monclient: monmap: 2024-03-25T18:22:17.530 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:22:17.530 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:17.530 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:22:17.530 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:22:17.530 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:17.530 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:22:17.530 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:22:17.530 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:22:17.530 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:22:17.530 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:17.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.527+0000 7f8ee204a640 1 librados: starting msgr at 2024-03-25T18:22:17.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.527+0000 7f8ee204a640 1 librados: starting objecter 2024-03-25T18:22:17.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.527+0000 7f8ee204a640 1 Processor -- start 2024-03-25T18:22:17.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.528+0000 7f8ee204a640 1 -- start start 2024-03-25T18:22:17.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.528+0000 7f8ee204a640 1 librados: setting wanted keys 2024-03-25T18:22:17.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.528+0000 7f8ee204a640 1 librados: calling monclient init 2024-03-25T18:22:17.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.528+0000 7f8ee204a640 10 monclient: init 2024-03-25T18:22:17.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.528+0000 7f8ee204a640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:17.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.528+0000 7f8ee204a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:17.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.528+0000 7f8ee204a640 10 monclient(hunting): picked mon.noname-b con 0x7f8edc0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:22:17.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.528+0000 7f8ee204a640 10 monclient(hunting): picked mon.noname-c con 0x7f8edc13f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:22:17.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.528+0000 7f8ee204a640 10 monclient(hunting): picked mon.noname-a con 0x7f8edc1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:22:17.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.528+0000 7f8ee204a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8edc136ec0 con 0x7f8edc1430c0 2024-03-25T18:22:17.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.528+0000 7f8ee204a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8edc136c20 con 0x7f8edc0a98f0 2024-03-25T18:22:17.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.529+0000 7f8ee204a640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8edc137db0 con 0x7f8edc13f710 2024-03-25T18:22:17.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.529+0000 7f8ee204a640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:17.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.529+0000 7f8ee204a640 10 monclient(hunting): authenticate will time out at 8031.715303s 2024-03-25T18:22:17.539 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.538+0000 7f8edaffd640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f8edc13f710 0x7f8edc0b0d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:53198/0 (socket says 172.21.15.63:53198) 2024-03-25T18:22:17.539 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.538+0000 7f8edaffd640 1 -- 172.21.15.63:0/1641847980 learned_addr learned my addr 172.21.15.63:0/1641847980 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:22:17.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.543+0000 7f8ec3fff640 10 client.?.objecter ms_handle_connect 0x7f8edc13f710 2024-03-25T18:22:17.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.543+0000 7f8ec3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:17.546 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.545+0000 7f8ec3fff640 10 client.?.objecter ms_handle_connect 0x7f8edc1430c0 2024-03-25T18:22:17.546 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.545+0000 7f8ec3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:17.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.556+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832765369 0 0) 0x7f8edc137db0 con 0x7f8edc13f710 2024-03-25T18:22:17.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.556+0000 7f8ec3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:17.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.556+0000 7f8ec3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:17.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.556+0000 7f8ec3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:17.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.556+0000 7f8ec3fff640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:22:17.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.556+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec8004eb0 con 0x7f8edc13f710 2024-03-25T18:22:17.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.557+0000 7f8ec3fff640 10 client.?.objecter ms_handle_connect 0x7f8edc0a98f0 2024-03-25T18:22:17.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.557+0000 7f8ec3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:17.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.560+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986091716 0 0) 0x7f8edc136ec0 con 0x7f8edc1430c0 2024-03-25T18:22:17.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.560+0000 7f8ec3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:17.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.560+0000 7f8ec3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:17.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.560+0000 7f8ec3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:17.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.560+0000 7f8ec3fff640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:22:17.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.560+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8edc137db0 con 0x7f8edc1430c0 2024-03-25T18:22:17.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.569+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3610454911 0 0) 0x7f8ec8004eb0 con 0x7f8edc13f710 2024-03-25T18:22:17.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.569+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8edc136ec0 con 0x7f8edc13f710 2024-03-25T18:22:17.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.569+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8ed0001a30 con 0x7f8edc13f710 2024-03-25T18:22:17.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.569+0000 7f8ec3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:17.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.569+0000 7f8ec3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:17.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.569+0000 7f8ec3fff640 10 monclient(hunting): dump: 2024-03-25T18:22:17.570 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:17.570 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:17.571 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:17.571 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:17.571 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:17.571 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:17.571 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:17.571 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:17.571 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:17.571 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:17.571 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.569+0000 7f8ec3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:17.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.572+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1075017945 0 0) 0x7f8edc137db0 con 0x7f8edc1430c0 2024-03-25T18:22:17.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.572+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ec8004eb0 con 0x7f8edc1430c0 2024-03-25T18:22:17.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.572+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8ed4001e80 con 0x7f8edc1430c0 2024-03-25T18:22:17.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.572+0000 7f8ec3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:17.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.572+0000 7f8ec3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:17.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.572+0000 7f8ec3fff640 10 monclient(hunting): dump: 2024-03-25T18:22:17.573 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:17.573 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:17.573 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:17.573 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:17.574 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:17.574 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:17.574 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:17.574 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:17.574 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:17.574 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:17.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.579+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536021155 0 0) 0x7f8edc136c20 con 0x7f8edc0a98f0 2024-03-25T18:22:17.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.579+0000 7f8ec3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:17.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.579+0000 7f8ec3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:17.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.579+0000 7f8ec3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:17.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.579+0000 7f8ec3fff640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:22:17.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.579+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8edc137db0 con 0x7f8edc0a98f0 2024-03-25T18:22:17.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.585+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 214855242 0 0) 0x7f8edc136ec0 con 0x7f8edc13f710 2024-03-25T18:22:17.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.585+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 >> v1:172.21.15.81:6789/0 conn(0x7f8edc0a98f0 legacy=0x7f8edc1376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:17.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.585+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 >> v1:172.21.15.63:6789/0 conn(0x7f8edc1430c0 legacy=0x7f8edc1e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:17.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.585+0000 7f8ec3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:17.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.585+0000 7f8ec3fff640 1 monclient: found mon.c 2024-03-25T18:22:17.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.585+0000 7f8ec3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:17.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.585+0000 7f8ec3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:22:17.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.585+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8edc138b50 con 0x7f8edc13f710 2024-03-25T18:22:17.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.585+0000 7f8ec3fff640 10 monclient: _finish_auth 0 2024-03-25T18:22:17.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.585+0000 7f8ec3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:17.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.585+0000 7f8ec3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:17.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.585+0000 7f8ee204a640 5 monclient: authenticate success, global_id 4652 2024-03-25T18:22:17.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.585+0000 7f8ee204a640 10 monclient: _renew_subs 2024-03-25T18:22:17.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.585+0000 7f8ee204a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:22:17.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.585+0000 7f8ee204a640 1 -- 172.21.15.63:0/1641847980 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8edc139830 con 0x7f8edc13f710 2024-03-25T18:22:17.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.586+0000 7f8ee204a640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:17.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.586+0000 7f8ee204a640 10 monclient: _renew_subs 2024-03-25T18:22:17.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.586+0000 7f8ee204a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:22:17.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.586+0000 7f8ee204a640 1 -- 172.21.15.63:0/1641847980 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8edc1369b0 con 0x7f8edc13f710 2024-03-25T18:22:17.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.586+0000 7f8ee204a640 1 librados: init done 2024-03-25T18:22:17.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.587+0000 7f8ee204a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:17.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.587+0000 7f8ee204a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:17.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.587+0000 7f8ee204a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:22:17.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.587+0000 7f8ee204a640 1 -- 172.21.15.63:0/1641847980 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8edc139a20 con 0x7f8edc13f710 2024-03-25T18:22:17.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.906+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ed0001c70 con 0x7f8edc13f710 2024-03-25T18:22:17.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.906+0000 7f8ec3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:17.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.906+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8ed00038c0 con 0x7f8edc13f710 2024-03-25T18:22:17.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.906+0000 7f8ec3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:17.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.906+0000 7f8ec3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:17.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.906+0000 7f8ec3fff640 10 monclient: dump: 2024-03-25T18:22:17.908 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:17.908 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:17.908 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:17.908 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:17.908 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:17.908 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:17.908 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:17.908 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:17.909 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:17.909 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:17.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.906+0000 7f8ed9ffb640 4 set_mon_vals no callback set 2024-03-25T18:22:17.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.907+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f8ed0003ae0 con 0x7f8edc13f710 2024-03-25T18:22:17.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.907+0000 7f8edb7fe640 10 monclient: get_auth_request con 0x7f8ec806df40 auth_method 0 2024-03-25T18:22:17.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.908+0000 7f8ec3fff640 10 client.4652.objecter ms_handle_connect 0x7f8ec806df40 2024-03-25T18:22:17.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.914+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f8ed00546b0 con 0x7f8edc13f710 2024-03-25T18:22:17.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.914+0000 7f8ec3fff640 10 client.4652.objecter ms_dispatch 0x7f8edc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:17.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.914+0000 7f8ec3fff640 3 client.4652.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:17.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.914+0000 7f8ec3fff640 3 client.4652.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:17.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.914+0000 7f8ec3fff640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-25T18:22:17.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.914+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ed00501c0 con 0x7f8edc13f710 2024-03-25T18:22:17.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.914+0000 7f8ec3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:17.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:17.914+0000 7f8ec3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:18.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.174+0000 7f8ee204a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:18.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.174+0000 7f8ee204a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:18.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.174+0000 7f8ee204a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:22:18.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.174+0000 7f8ee204a640 1 -- 172.21.15.63:0/1641847980 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8edc14f320 con 0x7f8edc13f710 2024-03-25T18:22:18.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.214+0000 7f8ec3fff640 1 -- 172.21.15.63:0/1641847980 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f8ed00520f0 con 0x7f8edc13f710 2024-03-25T18:22:18.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.214+0000 7f8ec3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:18.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.214+0000 7f8ec3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:18.216 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:22:18.216 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:18.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.217+0000 7f8ee204a640 10 librados: watch_flush enter 2024-03-25T18:22:18.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.217+0000 7f8ee204a640 10 librados: watch_flush exit 2024-03-25T18:22:18.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.217+0000 7f8ee204a640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-25T18:22:18.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.217+0000 7f8ee204a640 10 client.4652.objecter successfully canceled tick 2024-03-25T18:22:18.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.217+0000 7f8ee204a640 1 -- 172.21.15.63:0/1641847980 >> v1:172.21.15.81:6800/3640340873 conn(0x7f8ec806df40 legacy=0x7f8ec8070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:18.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.217+0000 7f8ee204a640 10 monclient: shutdown 2024-03-25T18:22:18.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.217+0000 7f8ee204a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:18.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.217+0000 7f8ee204a640 1 -- 172.21.15.63:0/1641847980 >> v1:172.21.15.63:6790/0 conn(0x7f8edc13f710 legacy=0x7f8edc0b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:18.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.217+0000 7f8ee204a640 1 -- 172.21.15.63:0/1641847980 shutdown_connections 2024-03-25T18:22:18.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.217+0000 7f8ee204a640 1 -- 172.21.15.63:0/1641847980 >> 172.21.15.63:0/1641847980 conn(0x7f8edc0b7ea0 msgr2=0x7f8edc13ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:18.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.217+0000 7f8ee204a640 1 -- 172.21.15.63:0/1641847980 shutdown_connections 2024-03-25T18:22:18.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.217+0000 7f8ee204a640 1 -- 172.21.15.63:0/1641847980 wait complete. 2024-03-25T18:22:18.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:18.217+0000 7f8ee204a640 1 librados: shutdown 2024-03-25T18:22:18.231 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:24.906 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:06.597 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:24.957 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:06.649 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:25.000 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:06.692 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:25.014 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:10:06.674 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:25.021 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:06.713 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:25.022 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:10:06.682 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:25.029 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:06.722 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:25.037 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:25.040 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:10:06.700 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:25.058 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:06.750 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:25.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.139+0000 7f20e732b640 10 monclient: get_monmap_and_config 2024-03-25T18:22:25.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.139+0000 7f20e732b640 10 monclient: build_initial_monmap 2024-03-25T18:22:25.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.140+0000 7f20e732b640 10 monclient: monmap: 2024-03-25T18:22:25.141 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:22:25.141 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:25.141 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:22:25.141116+0000 2024-03-25T18:22:25.141 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:22:25.141116+0000 2024-03-25T18:22:25.141 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:25.141 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:25.141 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:22:25.141 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:22:25.142 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:22:25.142 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:25.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.140+0000 7f20e732b640 1 Processor -- start 2024-03-25T18:22:25.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.140+0000 7f20e732b640 1 -- start start 2024-03-25T18:22:25.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.140+0000 7f20e732b640 10 monclient: init 2024-03-25T18:22:25.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.141+0000 7f20e732b640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:25.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.141+0000 7f20e732b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:25.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.141+0000 7f20e732b640 10 monclient(hunting): picked mon.noname-b con 0x7f20e00b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T18:22:25.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.141+0000 7f20e732b640 10 monclient(hunting): picked mon.noname-a con 0x7f20e00b1730 addr v1:172.21.15.63:6789/0 2024-03-25T18:22:25.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.141+0000 7f20e732b640 10 monclient(hunting): picked mon.noname-c con 0x7f20e014be90 addr v1:172.21.15.81:6789/0 2024-03-25T18:22:25.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.141+0000 7f20e732b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e00af940 con 0x7f20e00b1730 2024-03-25T18:22:25.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.141+0000 7f20e732b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e00b0a70 con 0x7f20e014be90 2024-03-25T18:22:25.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.141+0000 7f20e732b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e00b0d50 con 0x7f20e00b4d60 2024-03-25T18:22:25.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.141+0000 7f20e732b640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:25.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.141+0000 7f20e732b640 10 monclient(hunting): authenticate will time out at 8039.327827s 2024-03-25T18:22:25.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.172+0000 7f20e48c6640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f20e00b1730 0x7f20e00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:53416/0 (socket says 172.21.15.63:53416) 2024-03-25T18:22:25.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.172+0000 7f20e48c6640 1 -- 172.21.15.63:0/3685354722 learned_addr learned my addr 172.21.15.63:0/3685354722 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:22:25.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.184+0000 7f20d7fff640 1 -- 172.21.15.63:0/3685354722 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934481279 0 0) 0x7f20e00af940 con 0x7f20e00b1730 2024-03-25T18:22:25.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.184+0000 7f20d7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:25.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.184+0000 7f20d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:25.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.184+0000 7f20d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:25.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.184+0000 7f20d7fff640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:22:25.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.184+0000 7f20d7fff640 1 -- 172.21.15.63:0/3685354722 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20cc003520 con 0x7f20e00b1730 2024-03-25T18:22:25.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.195+0000 7f20d7fff640 1 -- 172.21.15.63:0/3685354722 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 597184730 0 0) 0x7f20cc003520 con 0x7f20e00b1730 2024-03-25T18:22:25.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20d7fff640 1 -- 172.21.15.63:0/3685354722 >> v1:172.21.15.63:6790/0 conn(0x7f20e00b4d60 legacy=0x7f20e013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:25.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20d7fff640 1 -- 172.21.15.63:0/3685354722 >> v1:172.21.15.81:6789/0 conn(0x7f20e014be90 legacy=0x7f20e014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:22:25.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20d7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:25.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20d7fff640 1 monclient: found mon.noname-a 2024-03-25T18:22:25.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:25.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20d7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:22:25.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20d7fff640 1 -- 172.21.15.63:0/3685354722 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e00b1030 con 0x7f20e00b1730 2024-03-25T18:22:25.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20d7fff640 10 monclient: _finish_auth 0 2024-03-25T18:22:25.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20d7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:25.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:25.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20d7fff640 1 -- 172.21.15.63:0/3685354722 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f20d0001690 con 0x7f20e00b1730 2024-03-25T18:22:25.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20e732b640 5 monclient: authenticate success, global_id 4641 2024-03-25T18:22:25.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20e732b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:25.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:25.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20d7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:25.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20d7fff640 10 monclient: dump: 2024-03-25T18:22:25.199 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:25.199 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:25.200 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:25.200 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:25.200 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:25.200 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:25.200 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:25.200 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:25.200 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:25.200 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:25.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.196+0000 7f20e732b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:25.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.218+0000 7f20d7fff640 1 -- 172.21.15.63:0/3685354722 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20d0001860 con 0x7f20e00b1730 2024-03-25T18:22:25.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.218+0000 7f20d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:25.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.218+0000 7f20d7fff640 1 -- 172.21.15.63:0/3685354722 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f20d0001cb0 con 0x7f20e00b1730 2024-03-25T18:22:25.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.218+0000 7f20d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:25.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.218+0000 7f20d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:25.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.218+0000 7f20d7fff640 10 monclient: dump: 2024-03-25T18:22:25.220 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:25.220 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:25.220 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:25.220 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:25.220 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:25.220 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:25.220 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:25.220 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:25.220 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:25.220 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:25.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.218+0000 7f20e732b640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:25.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.218+0000 7f20e732b640 4 set_mon_vals no callback set 2024-03-25T18:22:25.222 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.219+0000 7f20e732b640 10 monclient: shutdown 2024-03-25T18:22:25.222 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.219+0000 7f20e732b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:25.222 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.219+0000 7f20e732b640 1 -- 172.21.15.63:0/3685354722 >> v1:172.21.15.63:6789/0 conn(0x7f20e00b1730 legacy=0x7f20e00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:25.222 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.219+0000 7f20e732b640 1 -- 172.21.15.63:0/3685354722 shutdown_connections 2024-03-25T18:22:25.222 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.219+0000 7f20e732b640 1 -- 172.21.15.63:0/3685354722 >> 172.21.15.63:0/3685354722 conn(0x7f20e00617a0 msgr2=0x7f20e0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:25.222 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.219+0000 7f20e732b640 1 -- 172.21.15.63:0/3685354722 shutdown_connections 2024-03-25T18:22:25.222 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.219+0000 7f20e732b640 1 -- 172.21.15.63:0/3685354722 wait complete. 2024-03-25T18:22:25.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.219+0000 7f20e732b640 10 monclient: build_initial_monmap 2024-03-25T18:22:25.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.219+0000 7f20e732b640 10 monclient: monmap: 2024-03-25T18:22:25.223 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:22:25.223 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:25.223 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:22:25.223 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:22:25.223 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:25.223 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:22:25.223 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:22:25.223 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:22:25.223 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:22:25.223 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:25.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 1 librados: starting msgr at 2024-03-25T18:22:25.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 1 librados: starting objecter 2024-03-25T18:22:25.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 1 Processor -- start 2024-03-25T18:22:25.223 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 1 -- start start 2024-03-25T18:22:25.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 1 librados: setting wanted keys 2024-03-25T18:22:25.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 1 librados: calling monclient init 2024-03-25T18:22:25.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 10 monclient: init 2024-03-25T18:22:25.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:25.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:25.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 10 monclient(hunting): picked mon.noname-b con 0x7f20e00b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:22:25.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 10 monclient(hunting): picked mon.noname-c con 0x7f20e00b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T18:22:25.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 10 monclient(hunting): picked mon.noname-a con 0x7f20e014be90 addr v1:172.21.15.63:6789/0 2024-03-25T18:22:25.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e013a2c0 con 0x7f20e014be90 2024-03-25T18:22:25.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e013a510 con 0x7f20e00b1730 2024-03-25T18:22:25.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e0145810 con 0x7f20e00b4d60 2024-03-25T18:22:25.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:25.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.220+0000 7f20e732b640 10 monclient(hunting): authenticate will time out at 8039.407279s 2024-03-25T18:22:25.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.249+0000 7f20e50c7640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f20e00b1730 0x7f20e01df3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:57732/0 (socket says 172.21.15.63:57732) 2024-03-25T18:22:25.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.249+0000 7f20e50c7640 1 -- 172.21.15.63:0/1162481438 learned_addr learned my addr 172.21.15.63:0/1162481438 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:22:25.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.256+0000 7f20d5ffb640 10 client.?.objecter ms_handle_connect 0x7f20e00b1730 2024-03-25T18:22:25.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.256+0000 7f20d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:25.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.267+0000 7f20d5ffb640 1 -- 172.21.15.63:0/1162481438 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3797764192 0 0) 0x7f20e013a510 con 0x7f20e00b1730 2024-03-25T18:22:25.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.267+0000 7f20d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:25.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.267+0000 7f20d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:25.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.267+0000 7f20d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:25.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.267+0000 7f20d5ffb640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:22:25.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.267+0000 7f20d5ffb640 1 -- 172.21.15.63:0/1162481438 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20c4004eb0 con 0x7f20e00b1730 2024-03-25T18:22:25.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.269+0000 7f20d5ffb640 10 client.?.objecter ms_handle_connect 0x7f20e00b4d60 2024-03-25T18:22:25.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.269+0000 7f20d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:25.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.279+0000 7f20d5ffb640 1 -- 172.21.15.63:0/1162481438 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4213309321 0 0) 0x7f20c4004eb0 con 0x7f20e00b1730 2024-03-25T18:22:25.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.279+0000 7f20d5ffb640 1 -- 172.21.15.63:0/1162481438 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20e013a510 con 0x7f20e00b1730 2024-03-25T18:22:25.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.279+0000 7f20d5ffb640 1 -- 172.21.15.63:0/1162481438 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f20dc001a30 con 0x7f20e00b1730 2024-03-25T18:22:25.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.279+0000 7f20d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:25.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.279+0000 7f20d5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:25.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.279+0000 7f20d5ffb640 10 monclient(hunting): dump: 2024-03-25T18:22:25.281 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:25.281 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:25.281 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:25.281 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:25.281 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:25.281 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:25.282 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:25.282 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:25.282 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:25.282 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:25.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.279+0000 7f20d5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:25.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.285+0000 7f20d5ffb640 10 client.?.objecter ms_handle_connect 0x7f20e014be90 2024-03-25T18:22:25.287 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.285+0000 7f20d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:25.290 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20d5ffb640 1 -- 172.21.15.63:0/1162481438 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1110600432 0 0) 0x7f20e013a510 con 0x7f20e00b1730 2024-03-25T18:22:25.290 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20d5ffb640 1 -- 172.21.15.63:0/1162481438 >> v1:172.21.15.63:6790/0 conn(0x7f20e00b4d60 legacy=0x7f20e01dc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:25.290 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20d5ffb640 1 -- 172.21.15.63:0/1162481438 >> v1:172.21.15.63:6789/0 conn(0x7f20e014be90 legacy=0x7f20e01e7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:25.290 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20d5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:22:25.290 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20d5ffb640 1 monclient: found mon.b 2024-03-25T18:22:25.290 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:25.290 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20d5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:25.290 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20d5ffb640 1 -- 172.21.15.63:0/1162481438 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e0138dd0 con 0x7f20e00b1730 2024-03-25T18:22:25.290 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20d5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:22:25.290 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20d5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:22:25.290 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:25.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20e732b640 5 monclient: authenticate success, global_id 4681 2024-03-25T18:22:25.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20e732b640 10 monclient: _renew_subs 2024-03-25T18:22:25.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20e732b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:25.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20e732b640 1 -- 172.21.15.63:0/1162481438 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e0132470 con 0x7f20e00b1730 2024-03-25T18:22:25.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20e732b640 10 client.4681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:25.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20e732b640 10 monclient: _renew_subs 2024-03-25T18:22:25.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20e732b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:25.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20e732b640 1 -- 172.21.15.63:0/1162481438 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e0145410 con 0x7f20e00b1730 2024-03-25T18:22:25.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.288+0000 7f20e732b640 1 librados: init done 2024-03-25T18:22:25.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.290+0000 7f20e732b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:25.292 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.290+0000 7f20e732b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:25.292 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.290+0000 7f20e732b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:25.292 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.290+0000 7f20e732b640 1 -- 172.21.15.63:0/1162481438 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e00627b0 con 0x7f20e00b1730 2024-03-25T18:22:25.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.565+0000 7f20d5ffb640 1 -- 172.21.15.63:0/1162481438 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20dc001cf0 con 0x7f20e00b1730 2024-03-25T18:22:25.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.565+0000 7f20d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:25.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.565+0000 7f20d5ffb640 1 -- 172.21.15.63:0/1162481438 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f20dc003a60 con 0x7f20e00b1730 2024-03-25T18:22:25.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.565+0000 7f20d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:25.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.565+0000 7f20d5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:25.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.565+0000 7f20d5ffb640 10 monclient: dump: 2024-03-25T18:22:25.567 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:25.567 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:25.567 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:25.567 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:25.567 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:25.567 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:25.567 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:25.568 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:25.568 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:25.568 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:25.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.565+0000 7f20d77fe640 4 set_mon_vals no callback set 2024-03-25T18:22:25.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.566+0000 7f20d5ffb640 1 -- 172.21.15.63:0/1162481438 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f20dc003c80 con 0x7f20e00b1730 2024-03-25T18:22:25.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.567+0000 7f20e48c6640 10 monclient: get_auth_request con 0x7f20c406dd00 auth_method 0 2024-03-25T18:22:25.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.567+0000 7f20d5ffb640 10 client.4681.objecter ms_handle_connect 0x7f20c406dd00 2024-03-25T18:22:25.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.572+0000 7f20d5ffb640 1 -- 172.21.15.63:0/1162481438 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f20dc0548c0 con 0x7f20e00b1730 2024-03-25T18:22:25.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.572+0000 7f20d5ffb640 10 client.4681.objecter ms_dispatch 0x7f20e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:25.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.572+0000 7f20d5ffb640 3 client.4681.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:25.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.572+0000 7f20d5ffb640 3 client.4681.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:25.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.572+0000 7f20d5ffb640 20 client.4681.objecter dump_active .. 0 homeless 2024-03-25T18:22:25.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.574+0000 7f20d5ffb640 1 -- 172.21.15.63:0/1162481438 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20dc0503d0 con 0x7f20e00b1730 2024-03-25T18:22:25.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.574+0000 7f20d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:25.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.574+0000 7f20d5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:25.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.829+0000 7f20e732b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:25.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.829+0000 7f20e732b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:25.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.829+0000 7f20e732b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:25.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.829+0000 7f20e732b640 1 -- 172.21.15.63:0/1162481438 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20e0144e90 con 0x7f20e00b1730 2024-03-25T18:22:25.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.850+0000 7f20d5ffb640 1 -- 172.21.15.63:0/1162481438 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f20dc052300 con 0x7f20e00b1730 2024-03-25T18:22:25.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.850+0000 7f20d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:25.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.850+0000 7f20d5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:25.851 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:22:25.852 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:25.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.852+0000 7f20e732b640 10 librados: watch_flush enter 2024-03-25T18:22:25.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.852+0000 7f20e732b640 10 librados: watch_flush exit 2024-03-25T18:22:25.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.852+0000 7f20e732b640 20 client.4681.objecter shutdown clearing up homeless session... 2024-03-25T18:22:25.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.852+0000 7f20e732b640 10 client.4681.objecter successfully canceled tick 2024-03-25T18:22:25.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.852+0000 7f20e732b640 1 -- 172.21.15.63:0/1162481438 >> v1:172.21.15.81:6800/3640340873 conn(0x7f20c406dd00 legacy=0x7f20c40701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:25.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.852+0000 7f20e732b640 10 monclient: shutdown 2024-03-25T18:22:25.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.852+0000 7f20e732b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:25.855 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.852+0000 7f20e732b640 1 -- 172.21.15.63:0/1162481438 >> v1:172.21.15.81:6789/0 conn(0x7f20e00b1730 legacy=0x7f20e01df3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:25.855 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.852+0000 7f20e732b640 1 -- 172.21.15.63:0/1162481438 shutdown_connections 2024-03-25T18:22:25.855 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.852+0000 7f20e732b640 1 -- 172.21.15.63:0/1162481438 >> 172.21.15.63:0/1162481438 conn(0x7f20e00617a0 msgr2=0x7f20e0138440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:25.855 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.852+0000 7f20e732b640 1 -- 172.21.15.63:0/1162481438 shutdown_connections 2024-03-25T18:22:25.855 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.853+0000 7f20e732b640 1 -- 172.21.15.63:0/1162481438 wait complete. 2024-03-25T18:22:25.855 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:25.853+0000 7f20e732b640 1 librados: shutdown 2024-03-25T18:22:25.865 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:32.672 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:32.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.783+0000 7f4281063640 10 monclient: get_monmap_and_config 2024-03-25T18:22:32.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.783+0000 7f4281063640 10 monclient: build_initial_monmap 2024-03-25T18:22:32.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.783+0000 7f4281063640 10 monclient: monmap: 2024-03-25T18:22:32.784 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:22:32.784 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:32.784 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:22:32.784379+0000 2024-03-25T18:22:32.784 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:22:32.784379+0000 2024-03-25T18:22:32.785 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:32.785 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:32.785 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:22:32.785 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:22:32.785 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:22:32.785 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:32.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.784+0000 7f4281063640 1 Processor -- start 2024-03-25T18:22:32.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.784+0000 7f4281063640 1 -- start start 2024-03-25T18:22:32.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.784+0000 7f4281063640 10 monclient: init 2024-03-25T18:22:32.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.784+0000 7f4281063640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:32.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.784+0000 7f4281063640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:32.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.784+0000 7f4281063640 10 monclient(hunting): picked mon.noname-a con 0x7f427c0b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T18:22:32.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.784+0000 7f4281063640 10 monclient(hunting): picked mon.noname-c con 0x7f427c0b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:22:32.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.784+0000 7f4281063640 10 monclient(hunting): picked mon.noname-b con 0x7f427c14be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:22:32.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.784+0000 7f4281063640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427c0af940 con 0x7f427c0b4d60 2024-03-25T18:22:32.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.784+0000 7f4281063640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427c0b0a70 con 0x7f427c0b1730 2024-03-25T18:22:32.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.784+0000 7f4281063640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427c0b0d50 con 0x7f427c14be90 2024-03-25T18:22:32.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.784+0000 7f4281063640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:32.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.784+0000 7f4281063640 10 monclient(hunting): authenticate will time out at 8046.971130s 2024-03-25T18:22:32.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.797+0000 7f427a59c640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f427c0b1730 0x7f427c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:60628/0 (socket says 172.21.15.63:60628) 2024-03-25T18:22:32.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.797+0000 7f427a59c640 1 -- 172.21.15.63:0/1092147366 learned_addr learned my addr 172.21.15.63:0/1092147366 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:22:32.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.823+0000 7f4279d9b640 1 -- 172.21.15.63:0/1092147366 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026180414 0 0) 0x7f427c0b0a70 con 0x7f427c0b1730 2024-03-25T18:22:32.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.823+0000 7f4279d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:22:32.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.823+0000 7f4279d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:32.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.823+0000 7f4279d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:32.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.823+0000 7f4279d9b640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:22:32.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.823+0000 7f4279d9b640 1 -- 172.21.15.63:0/1092147366 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4264003520 con 0x7f427c0b1730 2024-03-25T18:22:32.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.829+0000 7f4279d9b640 1 -- 172.21.15.63:0/1092147366 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474848960 0 0) 0x7f427c0b0d50 con 0x7f427c14be90 2024-03-25T18:22:32.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.829+0000 7f4279d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:22:32.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.829+0000 7f4279d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:32.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.829+0000 7f4279d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:32.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.829+0000 7f4279d9b640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:22:32.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.829+0000 7f4279d9b640 1 -- 172.21.15.63:0/1092147366 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f427c0b0a70 con 0x7f427c14be90 2024-03-25T18:22:32.845 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.844+0000 7f4279d9b640 1 -- 172.21.15.63:0/1092147366 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223784833 0 0) 0x7f427c0af940 con 0x7f427c0b4d60 2024-03-25T18:22:32.845 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.844+0000 7f4279d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:22:32.845 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.844+0000 7f4279d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:32.845 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.844+0000 7f4279d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:32.845 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.844+0000 7f4279d9b640 10 monclient(hunting): my global_id is 4644 2024-03-25T18:22:32.845 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.844+0000 7f4279d9b640 1 -- 172.21.15.63:0/1092147366 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f427c0b0d50 con 0x7f427c0b4d60 2024-03-25T18:22:32.853 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4279d9b640 1 -- 172.21.15.63:0/1092147366 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3155060813 0 0) 0x7f4264003520 con 0x7f427c0b1730 2024-03-25T18:22:32.853 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4279d9b640 1 -- 172.21.15.63:0/1092147366 >> v1:172.21.15.63:6790/0 conn(0x7f427c14be90 legacy=0x7f427c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:32.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4279d9b640 1 -- 172.21.15.63:0/1092147366 >> v1:172.21.15.63:6789/0 conn(0x7f427c0b4d60 legacy=0x7f427c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:32.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4279d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:22:32.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4279d9b640 1 monclient: found mon.noname-c 2024-03-25T18:22:32.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4279d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:32.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4279d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:22:32.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4279d9b640 1 -- 172.21.15.63:0/1092147366 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c0b1030 con 0x7f427c0b1730 2024-03-25T18:22:32.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4279d9b640 10 monclient: _finish_auth 0 2024-03-25T18:22:32.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4279d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:22:32.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4279d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:32.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4279d9b640 1 -- 172.21.15.63:0/1092147366 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f4270001690 con 0x7f427c0b1730 2024-03-25T18:22:32.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4279d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:32.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4279d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:32.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4279d9b640 10 monclient: dump: 2024-03-25T18:22:32.854 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:32.854 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:32.855 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:32.855 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:32.855 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:32.855 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:32.855 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:32.855 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:32.855 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:32.855 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:32.855 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4281063640 5 monclient: authenticate success, global_id 4684 2024-03-25T18:22:32.855 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.852+0000 7f4281063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:32.875 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.874+0000 7f4279d9b640 1 -- 172.21.15.63:0/1092147366 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42700017f0 con 0x7f427c0b1730 2024-03-25T18:22:32.875 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.874+0000 7f4279d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:32.875 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.874+0000 7f4279d9b640 1 -- 172.21.15.63:0/1092147366 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f4270001c00 con 0x7f427c0b1730 2024-03-25T18:22:32.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.874+0000 7f4279d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:32.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.874+0000 7f4279d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:32.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.874+0000 7f4279d9b640 10 monclient: dump: 2024-03-25T18:22:32.876 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:32.876 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:32.876 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:32.876 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:32.876 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:32.876 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:32.876 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:32.876 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:32.876 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:32.876 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:32.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.874+0000 7f4281063640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:32.876 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.874+0000 7f4281063640 4 set_mon_vals no callback set 2024-03-25T18:22:32.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.875+0000 7f4281063640 10 monclient: shutdown 2024-03-25T18:22:32.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.875+0000 7f4281063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:32.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.875+0000 7f4281063640 1 -- 172.21.15.63:0/1092147366 >> v1:172.21.15.81:6789/0 conn(0x7f427c0b1730 legacy=0x7f427c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:32.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.875+0000 7f4281063640 1 -- 172.21.15.63:0/1092147366 shutdown_connections 2024-03-25T18:22:32.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.875+0000 7f4281063640 1 -- 172.21.15.63:0/1092147366 >> 172.21.15.63:0/1092147366 conn(0x7f427c0617a0 msgr2=0x7f427c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:32.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.875+0000 7f4281063640 1 -- 172.21.15.63:0/1092147366 shutdown_connections 2024-03-25T18:22:32.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.875+0000 7f4281063640 1 -- 172.21.15.63:0/1092147366 wait complete. 2024-03-25T18:22:32.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.876+0000 7f4281063640 10 monclient: build_initial_monmap 2024-03-25T18:22:32.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.876+0000 7f4281063640 10 monclient: monmap: 2024-03-25T18:22:32.878 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:22:32.878 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:32.879 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:22:32.879 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:22:32.879 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:32.879 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:22:32.879 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:22:32.879 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:22:32.879 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:22:32.879 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:32.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.876+0000 7f4281063640 1 librados: starting msgr at 2024-03-25T18:22:32.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.876+0000 7f4281063640 1 librados: starting objecter 2024-03-25T18:22:32.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.877+0000 7f4281063640 1 Processor -- start 2024-03-25T18:22:32.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.877+0000 7f4281063640 1 -- start start 2024-03-25T18:22:32.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.877+0000 7f4281063640 1 librados: setting wanted keys 2024-03-25T18:22:32.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.877+0000 7f4281063640 1 librados: calling monclient init 2024-03-25T18:22:32.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.877+0000 7f4281063640 10 monclient: init 2024-03-25T18:22:32.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.877+0000 7f4281063640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:32.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.877+0000 7f4281063640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:32.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.878+0000 7f4281063640 10 monclient(hunting): picked mon.noname-a con 0x7f427c0b1730 addr v1:172.21.15.63:6789/0 2024-03-25T18:22:32.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.878+0000 7f4281063640 10 monclient(hunting): picked mon.noname-b con 0x7f427c0b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T18:22:32.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.878+0000 7f4281063640 10 monclient(hunting): picked mon.noname-c con 0x7f427c14be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:22:32.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.878+0000 7f4281063640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427c134990 con 0x7f427c0b1730 2024-03-25T18:22:32.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.878+0000 7f4281063640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427c141b50 con 0x7f427c0b4d60 2024-03-25T18:22:32.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.878+0000 7f4281063640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427c141e30 con 0x7f427c14be90 2024-03-25T18:22:32.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.878+0000 7f4281063640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:32.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.878+0000 7f4281063640 10 monclient(hunting): authenticate will time out at 8047.064531s 2024-03-25T18:22:32.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.886+0000 7f427a59c640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f427c0b4d60 0x7f427c133b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:60716/0 (socket says 172.21.15.63:60716) 2024-03-25T18:22:32.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.886+0000 7f427a59c640 1 -- 172.21.15.63:0/3412443187 learned_addr learned my addr 172.21.15.63:0/3412443187 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:22:32.888 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.887+0000 7f42637fe640 10 client.?.objecter ms_handle_connect 0x7f427c0b4d60 2024-03-25T18:22:32.888 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.887+0000 7f42637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:32.900 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.899+0000 7f42637fe640 10 client.?.objecter ms_handle_connect 0x7f427c14be90 2024-03-25T18:22:32.900 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.899+0000 7f42637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:32.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.905+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147553805 0 0) 0x7f427c141b50 con 0x7f427c0b4d60 2024-03-25T18:22:32.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.905+0000 7f42637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:32.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.905+0000 7f42637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:32.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.905+0000 7f42637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:32.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.905+0000 7f42637fe640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:22:32.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.905+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4268004eb0 con 0x7f427c0b4d60 2024-03-25T18:22:32.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.907+0000 7f42637fe640 10 client.?.objecter ms_handle_connect 0x7f427c0b1730 2024-03-25T18:22:32.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.907+0000 7f42637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:32.914 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.912+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75642020 0 0) 0x7f427c141e30 con 0x7f427c14be90 2024-03-25T18:22:32.914 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.912+0000 7f42637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:32.914 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.912+0000 7f42637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:32.914 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.912+0000 7f42637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:32.914 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.912+0000 7f42637fe640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:22:32.914 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.912+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f427c141b50 con 0x7f427c14be90 2024-03-25T18:22:32.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.919+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1471180447 0 0) 0x7f4268004eb0 con 0x7f427c0b4d60 2024-03-25T18:22:32.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.920+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f427c141e30 con 0x7f427c0b4d60 2024-03-25T18:22:32.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.920+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f4270001bc0 con 0x7f427c0b4d60 2024-03-25T18:22:32.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.920+0000 7f42637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:32.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.920+0000 7f42637fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:32.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.920+0000 7f42637fe640 10 monclient(hunting): dump: 2024-03-25T18:22:32.921 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:32.921 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:32.921 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:32.922 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:32.922 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:32.922 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:32.922 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:32.922 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:32.922 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:32.922 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:32.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.920+0000 7f42637fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:32.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.926+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3653309427 0 0) 0x7f427c141b50 con 0x7f427c14be90 2024-03-25T18:22:32.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.926+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4268004eb0 con 0x7f427c14be90 2024-03-25T18:22:32.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.926+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f4274001b50 con 0x7f427c14be90 2024-03-25T18:22:32.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.926+0000 7f42637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:32.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.926+0000 7f42637fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:32.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.926+0000 7f42637fe640 10 monclient(hunting): dump: 2024-03-25T18:22:32.928 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:32.928 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:32.928 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:32.928 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:32.928 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:32.928 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:32.928 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:32.928 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:32.928 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:32.928 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:32.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 834765664 0 0) 0x7f427c141e30 con 0x7f427c0b4d60 2024-03-25T18:22:32.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 >> v1:172.21.15.63:6790/0 conn(0x7f427c14be90 legacy=0x7f427c134280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:32.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 >> v1:172.21.15.63:6789/0 conn(0x7f427c0b1730 legacy=0x7f427c142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:32.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f42637fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:32.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f42637fe640 1 monclient: found mon.b 2024-03-25T18:22:32.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f42637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:32.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f42637fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:32.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c1406d0 con 0x7f427c0b4d60 2024-03-25T18:22:32.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f42637fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:32.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f42637fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:32.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f42637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:32.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f4281063640 5 monclient: authenticate success, global_id 4687 2024-03-25T18:22:32.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f4281063640 10 monclient: _renew_subs 2024-03-25T18:22:32.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f4281063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:32.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f4281063640 1 -- 172.21.15.63:0/3412443187 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f427c140380 con 0x7f427c0b4d60 2024-03-25T18:22:32.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f4281063640 10 client.4687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:32.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f4281063640 10 monclient: _renew_subs 2024-03-25T18:22:32.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f4281063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:32.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f4281063640 1 -- 172.21.15.63:0/3412443187 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f427c13bb10 con 0x7f427c0b4d60 2024-03-25T18:22:32.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.927+0000 7f4281063640 1 librados: init done 2024-03-25T18:22:32.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.928+0000 7f4281063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:32.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.928+0000 7f4281063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:32.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.928+0000 7f4281063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:32.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:32.928+0000 7f4281063640 1 -- 172.21.15.63:0/3412443187 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f427c13bdc0 con 0x7f427c0b4d60 2024-03-25T18:22:33.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.205+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42700011f0 con 0x7f427c0b4d60 2024-03-25T18:22:33.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.205+0000 7f42637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:33.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.205+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f4270004cb0 con 0x7f427c0b4d60 2024-03-25T18:22:33.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.205+0000 7f42637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:33.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.205+0000 7f42637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:33.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.205+0000 7f42637fe640 10 monclient: dump: 2024-03-25T18:22:33.207 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:33.207 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:33.207 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:33.207 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:33.207 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:33.207 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:33.207 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:33.207 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:33.207 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:33.207 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:33.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.205+0000 7f427959a640 4 set_mon_vals no callback set 2024-03-25T18:22:33.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.206+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f427001ad00 con 0x7f427c0b4d60 2024-03-25T18:22:33.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.206+0000 7f427ad9d640 10 monclient: get_auth_request con 0x7f426806de80 auth_method 0 2024-03-25T18:22:33.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.207+0000 7f42637fe640 10 client.4687.objecter ms_handle_connect 0x7f426806de80 2024-03-25T18:22:33.214 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.213+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3589067872 0 0) 0x7f42700549d0 con 0x7f427c0b4d60 2024-03-25T18:22:33.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.213+0000 7f42637fe640 10 client.4687.objecter ms_dispatch 0x7f427c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:33.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.213+0000 7f42637fe640 3 client.4687.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:33.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.213+0000 7f42637fe640 3 client.4687.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:33.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.213+0000 7f42637fe640 20 client.4687.objecter dump_active .. 0 homeless 2024-03-25T18:22:33.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.215+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4270050310 con 0x7f427c0b4d60 2024-03-25T18:22:33.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.215+0000 7f42637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:33.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.215+0000 7f42637fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:33.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.478+0000 7f4281063640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:33.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.478+0000 7f4281063640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:33.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.478+0000 7f4281063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:22:33.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.478+0000 7f4281063640 1 -- 172.21.15.63:0/3412443187 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f427c140860 con 0x7f427c0b4d60 2024-03-25T18:22:33.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.637+0000 7f42637fe640 1 -- 172.21.15.63:0/3412443187 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2450146588) 0x7f4270052410 con 0x7f427c0b4d60 2024-03-25T18:22:33.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.637+0000 7f42637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:33.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.637+0000 7f42637fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:33.639 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:22:33.639 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:12:42.051123+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:12:42.051123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:33.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.639+0000 7f4281063640 10 librados: watch_flush enter 2024-03-25T18:22:33.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.639+0000 7f4281063640 10 librados: watch_flush exit 2024-03-25T18:22:33.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.639+0000 7f4281063640 20 client.4687.objecter shutdown clearing up homeless session... 2024-03-25T18:22:33.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.639+0000 7f4281063640 10 client.4687.objecter successfully canceled tick 2024-03-25T18:22:33.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.639+0000 7f4281063640 1 -- 172.21.15.63:0/3412443187 >> v1:172.21.15.81:6800/3640340873 conn(0x7f426806de80 legacy=0x7f4268070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:33.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.639+0000 7f4281063640 10 monclient: shutdown 2024-03-25T18:22:33.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.639+0000 7f4281063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:33.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.639+0000 7f4281063640 1 -- 172.21.15.63:0/3412443187 >> v1:172.21.15.81:6789/0 conn(0x7f427c0b4d60 legacy=0x7f427c133b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:33.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.639+0000 7f4281063640 1 -- 172.21.15.63:0/3412443187 shutdown_connections 2024-03-25T18:22:33.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.639+0000 7f4281063640 1 -- 172.21.15.63:0/3412443187 >> 172.21.15.63:0/3412443187 conn(0x7f427c0617a0 msgr2=0x7f427c13c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:33.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.639+0000 7f4281063640 1 -- 172.21.15.63:0/3412443187 shutdown_connections 2024-03-25T18:22:33.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.639+0000 7f4281063640 1 -- 172.21.15.63:0/3412443187 wait complete. 2024-03-25T18:22:33.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:33.640+0000 7f4281063640 1 librados: shutdown 2024-03-25T18:22:33.652 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:40.458 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:40.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.576+0000 7fa7f9b02640 10 monclient: get_monmap_and_config 2024-03-25T18:22:40.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.576+0000 7fa7f9b02640 10 monclient: build_initial_monmap 2024-03-25T18:22:40.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.576+0000 7fa7f9b02640 10 monclient: monmap: 2024-03-25T18:22:40.577 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:22:40.577 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:40.577 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:22:40.577268+0000 2024-03-25T18:22:40.577 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:22:40.577268+0000 2024-03-25T18:22:40.577 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:40.577 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:40.578 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:22:40.578 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:22:40.578 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:22:40.578 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:40.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.577+0000 7fa7f9b02640 1 Processor -- start 2024-03-25T18:22:40.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.577+0000 7fa7f9b02640 1 -- start start 2024-03-25T18:22:40.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.577+0000 7fa7f9b02640 10 monclient: init 2024-03-25T18:22:40.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.577+0000 7fa7f9b02640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:40.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.577+0000 7fa7f9b02640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:40.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.577+0000 7fa7f9b02640 10 monclient(hunting): picked mon.noname-a con 0x7fa7f40b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T18:22:40.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.577+0000 7fa7f9b02640 10 monclient(hunting): picked mon.noname-b con 0x7fa7f40b1730 addr v1:172.21.15.63:6790/0 2024-03-25T18:22:40.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.577+0000 7fa7f9b02640 10 monclient(hunting): picked mon.noname-c con 0x7fa7f414be90 addr v1:172.21.15.81:6789/0 2024-03-25T18:22:40.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.577+0000 7fa7f9b02640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f40af940 con 0x7fa7f40b4d60 2024-03-25T18:22:40.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.577+0000 7fa7f9b02640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f40b0a70 con 0x7fa7f414be90 2024-03-25T18:22:40.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.577+0000 7fa7f9b02640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f40b0d50 con 0x7fa7f40b1730 2024-03-25T18:22:40.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.577+0000 7fa7f9b02640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:40.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.577+0000 7fa7f9b02640 10 monclient(hunting): authenticate will time out at 8054.764245s 2024-03-25T18:22:40.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.589+0000 7fa7f2ffd640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fa7f40b1730 0x7fa7f40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:51454/0 (socket says 172.21.15.63:51454) 2024-03-25T18:22:40.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.590+0000 7fa7f2ffd640 1 -- 172.21.15.63:0/3096851797 learned_addr learned my addr 172.21.15.63:0/3096851797 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:22:40.595 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.594+0000 7fa7f27fc640 1 -- 172.21.15.63:0/3096851797 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510151782 0 0) 0x7fa7f40b0d50 con 0x7fa7f40b1730 2024-03-25T18:22:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.594+0000 7fa7f27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.594+0000 7fa7f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.594+0000 7fa7f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.594+0000 7fa7f27fc640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:22:40.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.594+0000 7fa7f27fc640 1 -- 172.21.15.63:0/3096851797 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7dc003520 con 0x7fa7f40b1730 2024-03-25T18:22:40.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f27fc640 1 -- 172.21.15.63:0/3096851797 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 243467852 0 0) 0x7fa7dc003520 con 0x7fa7f40b1730 2024-03-25T18:22:40.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f27fc640 1 -- 172.21.15.63:0/3096851797 >> v1:172.21.15.81:6789/0 conn(0x7fa7f414be90 legacy=0x7fa7f414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:40.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f27fc640 1 -- 172.21.15.63:0/3096851797 >> v1:172.21.15.63:6789/0 conn(0x7fa7f40b4d60 legacy=0x7fa7f413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:40.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:40.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f27fc640 1 monclient: found mon.noname-b 2024-03-25T18:22:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:22:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f27fc640 1 -- 172.21.15.63:0/3096851797 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f40b1030 con 0x7fa7f40b1730 2024-03-25T18:22:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f27fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f27fc640 1 -- 172.21.15.63:0/3096851797 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fa7e8001690 con 0x7fa7f40b1730 2024-03-25T18:22:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f9b02640 5 monclient: authenticate success, global_id 4667 2024-03-25T18:22:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f9b02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f27fc640 10 monclient: dump: 2024-03-25T18:22:40.615 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:40.615 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:40.615 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:40.616 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:40.616 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:40.616 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:40.616 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:40.616 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:40.616 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:40.616 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:40.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.613+0000 7fa7f9b02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:40.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.627+0000 7fa7f27fc640 1 -- 172.21.15.63:0/3096851797 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7e80017f0 con 0x7fa7f40b1730 2024-03-25T18:22:40.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.627+0000 7fa7f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:40.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.627+0000 7fa7f27fc640 1 -- 172.21.15.63:0/3096851797 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fa7e8001c00 con 0x7fa7f40b1730 2024-03-25T18:22:40.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.627+0000 7fa7f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:40.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.627+0000 7fa7f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:40.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.627+0000 7fa7f27fc640 10 monclient: dump: 2024-03-25T18:22:40.629 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:40.629 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:40.629 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:40.629 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:40.629 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:40.629 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:40.630 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:40.630 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:40.630 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:40.630 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:40.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.627+0000 7fa7f9b02640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:40.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.627+0000 7fa7f9b02640 4 set_mon_vals no callback set 2024-03-25T18:22:40.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.628+0000 7fa7f9b02640 10 monclient: shutdown 2024-03-25T18:22:40.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.628+0000 7fa7f9b02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:40.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.628+0000 7fa7f9b02640 1 -- 172.21.15.63:0/3096851797 >> v1:172.21.15.63:6790/0 conn(0x7fa7f40b1730 legacy=0x7fa7f40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:40.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.628+0000 7fa7f9b02640 1 -- 172.21.15.63:0/3096851797 shutdown_connections 2024-03-25T18:22:40.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.628+0000 7fa7f9b02640 1 -- 172.21.15.63:0/3096851797 >> 172.21.15.63:0/3096851797 conn(0x7fa7f40617a0 msgr2=0x7fa7f4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:40.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.628+0000 7fa7f9b02640 1 -- 172.21.15.63:0/3096851797 shutdown_connections 2024-03-25T18:22:40.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.628+0000 7fa7f9b02640 1 -- 172.21.15.63:0/3096851797 wait complete. 2024-03-25T18:22:40.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.629+0000 7fa7f9b02640 10 monclient: build_initial_monmap 2024-03-25T18:22:40.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.629+0000 7fa7f9b02640 10 monclient: monmap: 2024-03-25T18:22:40.633 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:22:40.633 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:40.633 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:22:40.633 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:22:40.633 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:40.633 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:22:40.633 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:22:40.633 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:22:40.633 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:22:40.633 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:40.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.629+0000 7fa7f9b02640 1 librados: starting msgr at 2024-03-25T18:22:40.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.629+0000 7fa7f9b02640 1 librados: starting objecter 2024-03-25T18:22:40.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.629+0000 7fa7f9b02640 1 Processor -- start 2024-03-25T18:22:40.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.629+0000 7fa7f9b02640 1 -- start start 2024-03-25T18:22:40.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.629+0000 7fa7f9b02640 1 librados: setting wanted keys 2024-03-25T18:22:40.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.629+0000 7fa7f9b02640 1 librados: calling monclient init 2024-03-25T18:22:40.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.629+0000 7fa7f9b02640 10 monclient: init 2024-03-25T18:22:40.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.630+0000 7fa7f9b02640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:40.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.630+0000 7fa7f9b02640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:40.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.630+0000 7fa7f9b02640 10 monclient(hunting): picked mon.noname-b con 0x7fa7f40b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:22:40.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.630+0000 7fa7f9b02640 10 monclient(hunting): picked mon.noname-a con 0x7fa7f40b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T18:22:40.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.630+0000 7fa7f9b02640 10 monclient(hunting): picked mon.noname-c con 0x7fa7f414be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:22:40.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.630+0000 7fa7f9b02640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f4141bf0 con 0x7fa7f40b4d60 2024-03-25T18:22:40.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.630+0000 7fa7f9b02640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f4141ed0 con 0x7fa7f40b1730 2024-03-25T18:22:40.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.630+0000 7fa7f9b02640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f41421b0 con 0x7fa7f414be90 2024-03-25T18:22:40.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.630+0000 7fa7f9b02640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:40.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.630+0000 7fa7f9b02640 10 monclient(hunting): authenticate will time out at 8054.816868s 2024-03-25T18:22:40.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.642+0000 7fa7f3fff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fa7f414be90 0x7fa7f41341f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:51494/0 (socket says 172.21.15.63:51494) 2024-03-25T18:22:40.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.642+0000 7fa7f3fff640 1 -- 172.21.15.63:0/1134671956 learned_addr learned my addr 172.21.15.63:0/1134671956 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:22:40.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.644+0000 7fa7dbfff640 10 client.?.objecter ms_handle_connect 0x7fa7f414be90 2024-03-25T18:22:40.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.644+0000 7fa7dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:40.647 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.646+0000 7fa7dbfff640 10 client.?.objecter ms_handle_connect 0x7fa7f40b4d60 2024-03-25T18:22:40.647 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.646+0000 7fa7dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:40.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.659+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544385796 0 0) 0x7fa7f41421b0 con 0x7fa7f414be90 2024-03-25T18:22:40.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.659+0000 7fa7dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:40.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.659+0000 7fa7dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:40.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.659+0000 7fa7dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:40.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.659+0000 7fa7dbfff640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:22:40.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.659+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7e0004eb0 con 0x7fa7f414be90 2024-03-25T18:22:40.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.662+0000 7fa7dbfff640 10 client.?.objecter ms_handle_connect 0x7fa7f40b1730 2024-03-25T18:22:40.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.662+0000 7fa7dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:40.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.666+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483685766 0 0) 0x7fa7f4141bf0 con 0x7fa7f40b4d60 2024-03-25T18:22:40.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.666+0000 7fa7dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:40.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.666+0000 7fa7dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:40.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.666+0000 7fa7dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:40.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.666+0000 7fa7dbfff640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:22:40.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.666+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7f41421b0 con 0x7fa7f40b4d60 2024-03-25T18:22:40.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.677+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 448389436 0 0) 0x7fa7f41421b0 con 0x7fa7f40b4d60 2024-03-25T18:22:40.679 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.677+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7f4141bf0 con 0x7fa7f40b4d60 2024-03-25T18:22:40.679 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.677+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fa7e8001b00 con 0x7fa7f40b4d60 2024-03-25T18:22:40.679 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.677+0000 7fa7dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:40.679 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.677+0000 7fa7dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:40.679 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.677+0000 7fa7dbfff640 10 monclient(hunting): dump: 2024-03-25T18:22:40.679 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:40.679 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:40.679 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:40.679 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:40.679 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:40.679 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:40.679 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:40.679 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:40.680 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:40.680 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:40.680 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.677+0000 7fa7dbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:40.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.684+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 256902297 0 0) 0x7fa7e0004eb0 con 0x7fa7f414be90 2024-03-25T18:22:40.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.684+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7f41421b0 con 0x7fa7f414be90 2024-03-25T18:22:40.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.684+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fa7ec001c90 con 0x7fa7f414be90 2024-03-25T18:22:40.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.684+0000 7fa7dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:40.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.684+0000 7fa7dbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:40.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.684+0000 7fa7dbfff640 10 monclient(hunting): dump: 2024-03-25T18:22:40.686 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:40.686 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:40.686 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:40.686 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:40.686 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:40.686 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:40.686 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:40.686 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:40.686 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:40.686 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:40.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.685+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102884473 0 0) 0x7fa7f4141ed0 con 0x7fa7f40b1730 2024-03-25T18:22:40.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.685+0000 7fa7dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:40.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.685+0000 7fa7dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:40.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.685+0000 7fa7dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:40.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.685+0000 7fa7dbfff640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:22:40.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.685+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7e0004eb0 con 0x7fa7f40b1730 2024-03-25T18:22:40.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 49657797 0 0) 0x7fa7f4141bf0 con 0x7fa7f40b4d60 2024-03-25T18:22:40.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 >> v1:172.21.15.63:6790/0 conn(0x7fa7f414be90 legacy=0x7fa7f41341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:40.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 >> v1:172.21.15.81:6789/0 conn(0x7fa7f40b1730 legacy=0x7fa7f41425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:40.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7dbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:40.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7dbfff640 1 monclient: found mon.a 2024-03-25T18:22:40.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:40.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7dbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:22:40.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f4140f00 con 0x7fa7f40b4d60 2024-03-25T18:22:40.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7dbfff640 10 monclient: _finish_auth 0 2024-03-25T18:22:40.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7dbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:40.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:40.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7f9b02640 5 monclient: authenticate success, global_id 4653 2024-03-25T18:22:40.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7f9b02640 10 monclient: _renew_subs 2024-03-25T18:22:40.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7f9b02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:22:40.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7f9b02640 1 -- 172.21.15.63:0/1134671956 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f41406f0 con 0x7fa7f40b4d60 2024-03-25T18:22:40.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7f9b02640 10 client.4653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:40.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7f9b02640 10 monclient: _renew_subs 2024-03-25T18:22:40.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7f9b02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:22:40.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7f9b02640 1 -- 172.21.15.63:0/1134671956 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f4140380 con 0x7fa7f40b4d60 2024-03-25T18:22:40.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.698+0000 7fa7f9b02640 1 librados: init done 2024-03-25T18:22:40.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.699+0000 7fa7f9b02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:40.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.699+0000 7fa7f9b02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:40.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.699+0000 7fa7f9b02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:22:40.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:40.699+0000 7fa7f9b02640 1 -- 172.21.15.63:0/1134671956 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f41401f0 con 0x7fa7f40b4d60 2024-03-25T18:22:41.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.007+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7e8001ca0 con 0x7fa7f40b4d60 2024-03-25T18:22:41.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.007+0000 7fa7dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:41.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.007+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fa7e8004c30 con 0x7fa7f40b4d60 2024-03-25T18:22:41.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.007+0000 7fa7dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:41.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.007+0000 7fa7dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:41.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.007+0000 7fa7dbfff640 10 monclient: dump: 2024-03-25T18:22:41.009 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:41.009 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:41.009 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:41.009 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:41.009 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:41.009 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:41.009 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:41.009 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:41.009 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:41.009 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:41.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.007+0000 7fa7f1ffb640 4 set_mon_vals no callback set 2024-03-25T18:22:41.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.007+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fa7e801ac00 con 0x7fa7f40b4d60 2024-03-25T18:22:41.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.008+0000 7fa7f37fe640 10 monclient: get_auth_request con 0x7fa7e006df50 auth_method 0 2024-03-25T18:22:41.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.008+0000 7fa7dbfff640 10 client.4653.objecter ms_handle_connect 0x7fa7e006df50 2024-03-25T18:22:41.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.015+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2076230003 0 0) 0x7fa7e8054800 con 0x7fa7f40b4d60 2024-03-25T18:22:41.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.015+0000 7fa7dbfff640 10 client.4653.objecter ms_dispatch 0x7fa7f4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:22:41.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.015+0000 7fa7dbfff640 3 client.4653.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:22:41.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.015+0000 7fa7dbfff640 3 client.4653.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:22:41.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.015+0000 7fa7dbfff640 20 client.4653.objecter dump_active .. 0 homeless 2024-03-25T18:22:41.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.015+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7e8050140 con 0x7fa7f40b4d60 2024-03-25T18:22:41.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.015+0000 7fa7dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:41.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.015+0000 7fa7dbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:41.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.235+0000 7fa7f9b02640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:41.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.235+0000 7fa7f9b02640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:41.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.235+0000 7fa7f9b02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:22:41.237 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.235+0000 7fa7f9b02640 1 -- 172.21.15.63:0/1134671956 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7f40627b0 con 0x7fa7f40b4d60 2024-03-25T18:22:41.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.266+0000 7fa7dbfff640 1 -- 172.21.15.63:0/1134671956 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 515194931) 0x7fa7e8052240 con 0x7fa7f40b4d60 2024-03-25T18:22:41.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.266+0000 7fa7dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:41.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.266+0000 7fa7dbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:41.267 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:22:41.268 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":11,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:22:39.912010+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:22:39.912010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:41.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.268+0000 7fa7f9b02640 10 librados: watch_flush enter 2024-03-25T18:22:41.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.268+0000 7fa7f9b02640 10 librados: watch_flush exit 2024-03-25T18:22:41.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.268+0000 7fa7f9b02640 20 client.4653.objecter shutdown clearing up homeless session... 2024-03-25T18:22:41.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.268+0000 7fa7f9b02640 10 client.4653.objecter successfully canceled tick 2024-03-25T18:22:41.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.269+0000 7fa7f9b02640 1 -- 172.21.15.63:0/1134671956 >> v1:172.21.15.81:6800/3640340873 conn(0x7fa7e006df50 legacy=0x7fa7e0070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:41.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.269+0000 7fa7f9b02640 10 monclient: shutdown 2024-03-25T18:22:41.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.269+0000 7fa7f9b02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:41.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.269+0000 7fa7f9b02640 1 -- 172.21.15.63:0/1134671956 >> v1:172.21.15.63:6789/0 conn(0x7fa7f40b4d60 legacy=0x7fa7f4133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:41.271 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.269+0000 7fa7f9b02640 1 -- 172.21.15.63:0/1134671956 shutdown_connections 2024-03-25T18:22:41.271 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.269+0000 7fa7f9b02640 1 -- 172.21.15.63:0/1134671956 >> 172.21.15.63:0/1134671956 conn(0x7fa7f40617a0 msgr2=0x7fa7f413c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:41.271 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.269+0000 7fa7f9b02640 1 -- 172.21.15.63:0/1134671956 shutdown_connections 2024-03-25T18:22:41.271 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.269+0000 7fa7f9b02640 1 -- 172.21.15.63:0/1134671956 wait complete. 2024-03-25T18:22:41.271 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:41.269+0000 7fa7f9b02640 1 librados: shutdown 2024-03-25T18:22:41.282 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:44.985 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:26.677 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:45.038 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:26.730 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:45.069 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:26.761 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:45.087 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:10:26.748 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:45.100 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:10:26.760 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:45.112 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:26.804 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:45.121 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:26.813 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:45.127 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:10:26.787 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:45.149 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:26.841 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:48.088 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:48.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.204+0000 7f4558ee6640 10 monclient: get_monmap_and_config 2024-03-25T18:22:48.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.204+0000 7f4558ee6640 10 monclient: build_initial_monmap 2024-03-25T18:22:48.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.204+0000 7f4558ee6640 10 monclient: monmap: 2024-03-25T18:22:48.206 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:22:48.206 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:48.206 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:22:48.205787+0000 2024-03-25T18:22:48.206 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:22:48.205787+0000 2024-03-25T18:22:48.206 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:48.206 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:48.206 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:22:48.206 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:22:48.206 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:22:48.206 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:48.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.205+0000 7f4558ee6640 1 Processor -- start 2024-03-25T18:22:48.207 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.205+0000 7f4558ee6640 1 -- start start 2024-03-25T18:22:48.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.205+0000 7f4558ee6640 10 monclient: init 2024-03-25T18:22:48.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.206+0000 7f4558ee6640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:48.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.206+0000 7f4558ee6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:48.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.206+0000 7f4558ee6640 10 monclient(hunting): picked mon.noname-a con 0x7f45540a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:22:48.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.206+0000 7f4558ee6640 10 monclient(hunting): picked mon.noname-b con 0x7f455413f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:22:48.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.206+0000 7f4558ee6640 10 monclient(hunting): picked mon.noname-c con 0x7f45541430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:22:48.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.206+0000 7f4558ee6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4554142520 con 0x7f45540a98f0 2024-03-25T18:22:48.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.206+0000 7f4558ee6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45540b28a0 con 0x7f45541430c0 2024-03-25T18:22:48.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.206+0000 7f4558ee6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45540b2a30 con 0x7f455413f710 2024-03-25T18:22:48.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.206+0000 7f4558ee6640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:48.208 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.206+0000 7f4558ee6640 10 monclient(hunting): authenticate will time out at 8062.392649s 2024-03-25T18:22:48.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.214+0000 7f4552d9d640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f45541430c0 0x7f455414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:54556/0 (socket says 172.21.15.63:54556) 2024-03-25T18:22:48.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.214+0000 7f4552d9d640 1 -- 172.21.15.63:0/3359566866 learned_addr learned my addr 172.21.15.63:0/3359566866 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:22:48.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.218+0000 7f455159a640 1 -- 172.21.15.63:0/3359566866 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675706936 0 0) 0x7f45540b28a0 con 0x7f45541430c0 2024-03-25T18:22:48.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.218+0000 7f455159a640 10 monclient(hunting): _negotiate 2024-03-25T18:22:48.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.218+0000 7f455159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:48.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.218+0000 7f455159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:48.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.218+0000 7f455159a640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:22:48.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.218+0000 7f455159a640 1 -- 172.21.15.63:0/3359566866 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f453c003520 con 0x7f45541430c0 2024-03-25T18:22:48.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f455159a640 1 -- 172.21.15.63:0/3359566866 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1433074975 0 0) 0x7f453c003520 con 0x7f45541430c0 2024-03-25T18:22:48.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f455159a640 1 -- 172.21.15.63:0/3359566866 >> v1:172.21.15.63:6790/0 conn(0x7f455413f710 legacy=0x7f45541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:48.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f455159a640 1 -- 172.21.15.63:0/3359566866 >> v1:172.21.15.63:6789/0 conn(0x7f45540a98f0 legacy=0x7f45540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:48.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f455159a640 10 monclient: _finish_hunting 0 2024-03-25T18:22:48.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f455159a640 1 monclient: found mon.noname-c 2024-03-25T18:22:48.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f455159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:48.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f455159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:22:48.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f455159a640 1 -- 172.21.15.63:0/3359566866 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45540b44c0 con 0x7f45541430c0 2024-03-25T18:22:48.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f455159a640 10 monclient: _finish_auth 0 2024-03-25T18:22:48.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f455159a640 10 monclient: _check_auth_tickets 2024-03-25T18:22:48.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f455159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:48.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f455159a640 1 -- 172.21.15.63:0/3359566866 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f454c001a10 con 0x7f45541430c0 2024-03-25T18:22:48.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f455159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:48.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f455159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:48.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f455159a640 10 monclient: dump: 2024-03-25T18:22:48.229 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:48.229 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:48.229 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:48.229 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:48.229 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:48.229 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:48.229 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:48.229 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:48.229 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:48.230 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:48.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f4558ee6640 5 monclient: authenticate success, global_id 4696 2024-03-25T18:22:48.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.226+0000 7f4558ee6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:48.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.237+0000 7f455159a640 1 -- 172.21.15.63:0/3359566866 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f454c001bf0 con 0x7f45541430c0 2024-03-25T18:22:48.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.237+0000 7f455159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:48.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.237+0000 7f455159a640 1 -- 172.21.15.63:0/3359566866 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f454c002040 con 0x7f45541430c0 2024-03-25T18:22:48.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.237+0000 7f455159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:48.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.237+0000 7f455159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:48.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.237+0000 7f455159a640 10 monclient: dump: 2024-03-25T18:22:48.239 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:48.239 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:48.239 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:48.239 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:48.239 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:48.239 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:48.239 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:48.239 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:48.240 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:48.240 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:48.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.237+0000 7f4558ee6640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:48.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.237+0000 7f4558ee6640 4 set_mon_vals no callback set 2024-03-25T18:22:48.241 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.238+0000 7f4558ee6640 10 monclient: shutdown 2024-03-25T18:22:48.241 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.238+0000 7f4558ee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:48.241 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.238+0000 7f4558ee6640 1 -- 172.21.15.63:0/3359566866 >> v1:172.21.15.81:6789/0 conn(0x7f45541430c0 legacy=0x7f455414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:48.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.238+0000 7f4558ee6640 1 -- 172.21.15.63:0/3359566866 shutdown_connections 2024-03-25T18:22:48.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.238+0000 7f4558ee6640 1 -- 172.21.15.63:0/3359566866 >> 172.21.15.63:0/3359566866 conn(0x7f45540b7ea0 msgr2=0x7f45540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:48.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.238+0000 7f4558ee6640 1 -- 172.21.15.63:0/3359566866 shutdown_connections 2024-03-25T18:22:48.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.238+0000 7f4558ee6640 1 -- 172.21.15.63:0/3359566866 wait complete. 2024-03-25T18:22:48.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.239+0000 7f4558ee6640 10 monclient: build_initial_monmap 2024-03-25T18:22:48.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.239+0000 7f4558ee6640 10 monclient: monmap: 2024-03-25T18:22:48.242 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:22:48.242 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:48.242 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:22:48.242 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:22:48.242 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:48.242 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:22:48.242 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:22:48.243 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:22:48.243 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:22:48.243 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:48.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.240+0000 7f4558ee6640 1 librados: starting msgr at 2024-03-25T18:22:48.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.240+0000 7f4558ee6640 1 librados: starting objecter 2024-03-25T18:22:48.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.240+0000 7f4558ee6640 1 Processor -- start 2024-03-25T18:22:48.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.240+0000 7f4558ee6640 1 -- start start 2024-03-25T18:22:48.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.240+0000 7f4558ee6640 1 librados: setting wanted keys 2024-03-25T18:22:48.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.240+0000 7f4558ee6640 1 librados: calling monclient init 2024-03-25T18:22:48.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.240+0000 7f4558ee6640 10 monclient: init 2024-03-25T18:22:48.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.241+0000 7f4558ee6640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:48.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.241+0000 7f4558ee6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:48.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.241+0000 7f4558ee6640 10 monclient(hunting): picked mon.noname-b con 0x7f45540a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:22:48.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.241+0000 7f4558ee6640 10 monclient(hunting): picked mon.noname-a con 0x7f455413f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:22:48.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.241+0000 7f4558ee6640 10 monclient(hunting): picked mon.noname-c con 0x7f45541430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:22:48.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.241+0000 7f4558ee6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4554137140 con 0x7f455413f710 2024-03-25T18:22:48.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.241+0000 7f4558ee6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45541d7b90 con 0x7f45540a98f0 2024-03-25T18:22:48.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.241+0000 7f4558ee6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45541d7590 con 0x7f45541430c0 2024-03-25T18:22:48.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.241+0000 7f4558ee6640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:48.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.241+0000 7f4558ee6640 10 monclient(hunting): authenticate will time out at 8062.428042s 2024-03-25T18:22:48.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.259+0000 7f4552d9d640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f45541430c0 0x7f45541e1fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:56036/0 (socket says 172.21.15.63:56036) 2024-03-25T18:22:48.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.259+0000 7f4552d9d640 1 -- 172.21.15.63:0/4145611455 learned_addr learned my addr 172.21.15.63:0/4145611455 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:22:48.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.260+0000 7f4542ffd640 10 client.?.objecter ms_handle_connect 0x7f45541430c0 2024-03-25T18:22:48.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.260+0000 7f4542ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:48.264 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.263+0000 7f4542ffd640 10 client.?.objecter ms_handle_connect 0x7f455413f710 2024-03-25T18:22:48.264 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.263+0000 7f4542ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:48.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.267+0000 7f4542ffd640 10 client.?.objecter ms_handle_connect 0x7f45540a98f0 2024-03-25T18:22:48.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.267+0000 7f4542ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:48.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.272+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728139420 0 0) 0x7f4554137140 con 0x7f455413f710 2024-03-25T18:22:48.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.272+0000 7f4542ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:48.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.272+0000 7f4542ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:48.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.272+0000 7f4542ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:48.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.272+0000 7f4542ffd640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:22:48.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.272+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4534004eb0 con 0x7f455413f710 2024-03-25T18:22:48.276 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.275+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305761298 0 0) 0x7f45541d7590 con 0x7f45541430c0 2024-03-25T18:22:48.276 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.275+0000 7f4542ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:48.276 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.275+0000 7f4542ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:48.276 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.275+0000 7f4542ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:48.276 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.275+0000 7f4542ffd640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:22:48.276 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.275+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4554137140 con 0x7f45541430c0 2024-03-25T18:22:48.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.282+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260886003 0 0) 0x7f45541d7b90 con 0x7f45540a98f0 2024-03-25T18:22:48.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.282+0000 7f4542ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:48.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.282+0000 7f4542ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:48.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.282+0000 7f4542ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:48.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.282+0000 7f4542ffd640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:22:48.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.282+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45541d7590 con 0x7f45540a98f0 2024-03-25T18:22:48.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.306+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 484883266 0 0) 0x7f45541d7590 con 0x7f45540a98f0 2024-03-25T18:22:48.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.306+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45541d7b90 con 0x7f45540a98f0 2024-03-25T18:22:48.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.306+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f4544001830 con 0x7f45540a98f0 2024-03-25T18:22:48.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.306+0000 7f4542ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:48.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.306+0000 7f4542ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:48.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.306+0000 7f4542ffd640 10 monclient(hunting): dump: 2024-03-25T18:22:48.307 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:48.308 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:48.308 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:48.308 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:48.308 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:48.308 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:48.308 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:48.308 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:48.308 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:48.308 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:48.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.306+0000 7f4542ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:48.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.308+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 892168425 0 0) 0x7f4534004eb0 con 0x7f455413f710 2024-03-25T18:22:48.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.308+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45541d7590 con 0x7f455413f710 2024-03-25T18:22:48.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.308+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f45480018c0 con 0x7f455413f710 2024-03-25T18:22:48.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.308+0000 7f4542ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:48.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.308+0000 7f4542ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:48.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.308+0000 7f4542ffd640 10 monclient(hunting): dump: 2024-03-25T18:22:48.309 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:48.309 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:48.310 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:48.310 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:48.310 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:48.310 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:48.310 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:48.310 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:48.310 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:48.310 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:48.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.310+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3064000318 0 0) 0x7f4554137140 con 0x7f45541430c0 2024-03-25T18:22:48.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.310+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4534004eb0 con 0x7f45541430c0 2024-03-25T18:22:48.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.310+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f454c001f00 con 0x7f45541430c0 2024-03-25T18:22:48.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.310+0000 7f4542ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:48.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.310+0000 7f4542ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:48.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.310+0000 7f4542ffd640 10 monclient(hunting): dump: 2024-03-25T18:22:48.312 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:48.312 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:48.312 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:48.312 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:48.312 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:48.312 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:48.312 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:48.312 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:48.312 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:48.312 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:48.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 747277402 0 0) 0x7f45541d7590 con 0x7f455413f710 2024-03-25T18:22:48.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 >> v1:172.21.15.63:6790/0 conn(0x7f45541430c0 legacy=0x7f45541e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:48.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 >> v1:172.21.15.81:6789/0 conn(0x7f45540a98f0 legacy=0x7f45541394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:48.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4542ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:48.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4542ffd640 1 monclient: found mon.a 2024-03-25T18:22:48.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4542ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:48.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4542ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:22:48.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45541d7720 con 0x7f455413f710 2024-03-25T18:22:48.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4542ffd640 10 monclient: _finish_auth 0 2024-03-25T18:22:48.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4542ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:48.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4542ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:48.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4558ee6640 5 monclient: authenticate success, global_id 4659 2024-03-25T18:22:48.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4558ee6640 10 monclient: _renew_subs 2024-03-25T18:22:48.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4558ee6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:22:48.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4558ee6640 1 -- 172.21.15.63:0/4145611455 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455413b7d0 con 0x7f455413f710 2024-03-25T18:22:48.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4558ee6640 10 client.4659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:48.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4558ee6640 10 monclient: _renew_subs 2024-03-25T18:22:48.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4558ee6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:22:48.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4558ee6640 1 -- 172.21.15.63:0/4145611455 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45541d8290 con 0x7f455413f710 2024-03-25T18:22:48.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.341+0000 7f4558ee6640 1 librados: init done 2024-03-25T18:22:48.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.342+0000 7f4558ee6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:48.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.342+0000 7f4558ee6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:48.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.342+0000 7f4558ee6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:22:48.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.342+0000 7f4558ee6640 1 -- 172.21.15.63:0/4145611455 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f455413bb70 con 0x7f455413f710 2024-03-25T18:22:48.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.674+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4548001a20 con 0x7f455413f710 2024-03-25T18:22:48.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.674+0000 7f4542ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:48.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.674+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f4548003810 con 0x7f455413f710 2024-03-25T18:22:48.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.674+0000 7f4542ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:48.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.674+0000 7f4542ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:48.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.674+0000 7f4542ffd640 10 monclient: dump: 2024-03-25T18:22:48.676 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:48.676 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:48.677 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:48.677 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:48.677 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:48.677 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:48.677 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:48.677 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:48.677 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:48.677 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:48.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.674+0000 7f4550d99640 4 set_mon_vals no callback set 2024-03-25T18:22:48.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.675+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f45480198a0 con 0x7f455413f710 2024-03-25T18:22:48.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.676+0000 7f455259c640 10 monclient: get_auth_request con 0x7f453406e0c0 auth_method 0 2024-03-25T18:22:48.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.676+0000 7f4542ffd640 10 client.4659.objecter ms_handle_connect 0x7f453406e0c0 2024-03-25T18:22:48.681 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.680+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2076230003 0 0) 0x7f45480535e0 con 0x7f455413f710 2024-03-25T18:22:48.681 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.680+0000 7f4542ffd640 10 client.4659.objecter ms_dispatch 0x7f4554000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:22:48.681 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.680+0000 7f4542ffd640 3 client.4659.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:22:48.681 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.680+0000 7f4542ffd640 3 client.4659.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:22:48.681 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.680+0000 7f4542ffd640 20 client.4659.objecter dump_active .. 0 homeless 2024-03-25T18:22:48.681 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.680+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f454804ef20 con 0x7f455413f710 2024-03-25T18:22:48.681 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.680+0000 7f4542ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:48.681 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.680+0000 7f4542ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:48.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.926+0000 7f4558ee6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:48.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.926+0000 7f4558ee6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:48.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.926+0000 7f4558ee6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:22:48.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.926+0000 7f4558ee6640 1 -- 172.21.15.63:0/4145611455 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45541d7d20 con 0x7f455413f710 2024-03-25T18:22:48.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.969+0000 7f4542ffd640 1 -- 172.21.15.63:0/4145611455 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 515194931) 0x7f4548051020 con 0x7f455413f710 2024-03-25T18:22:48.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.969+0000 7f4542ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:48.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.969+0000 7f4542ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:48.971 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:22:48.971 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":11,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:22:39.912010+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:22:39.912010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:48.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.972+0000 7f4558ee6640 10 librados: watch_flush enter 2024-03-25T18:22:48.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.972+0000 7f4558ee6640 10 librados: watch_flush exit 2024-03-25T18:22:48.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.972+0000 7f4558ee6640 20 client.4659.objecter shutdown clearing up homeless session... 2024-03-25T18:22:48.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.972+0000 7f4558ee6640 10 client.4659.objecter successfully canceled tick 2024-03-25T18:22:48.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.972+0000 7f4558ee6640 1 -- 172.21.15.63:0/4145611455 >> v1:172.21.15.81:6800/3640340873 conn(0x7f453406e0c0 legacy=0x7f4534070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:48.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.972+0000 7f4558ee6640 10 monclient: shutdown 2024-03-25T18:22:48.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.972+0000 7f4558ee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:48.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.972+0000 7f4558ee6640 1 -- 172.21.15.63:0/4145611455 >> v1:172.21.15.63:6789/0 conn(0x7f455413f710 legacy=0x7f4554136a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:48.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.972+0000 7f4558ee6640 1 -- 172.21.15.63:0/4145611455 shutdown_connections 2024-03-25T18:22:48.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.972+0000 7f4558ee6640 1 -- 172.21.15.63:0/4145611455 >> 172.21.15.63:0/4145611455 conn(0x7f45540b7ea0 msgr2=0x7f4554074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:48.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.972+0000 7f4558ee6640 1 -- 172.21.15.63:0/4145611455 shutdown_connections 2024-03-25T18:22:48.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.972+0000 7f4558ee6640 1 -- 172.21.15.63:0/4145611455 wait complete. 2024-03-25T18:22:48.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:48.972+0000 7f4558ee6640 1 librados: shutdown 2024-03-25T18:22:48.986 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:55.793 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:55.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.891+0000 7f8cd432b640 10 monclient: get_monmap_and_config 2024-03-25T18:22:55.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.891+0000 7f8cd432b640 10 monclient: build_initial_monmap 2024-03-25T18:22:55.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.891+0000 7f8cd432b640 10 monclient: monmap: 2024-03-25T18:22:55.892 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:22:55.892 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:55.892 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:22:55.892444+0000 2024-03-25T18:22:55.892 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:22:55.892444+0000 2024-03-25T18:22:55.893 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:55.893 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:55.893 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:22:55.893 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:22:55.893 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:22:55.893 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:55.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.891+0000 7f8cd432b640 1 Processor -- start 2024-03-25T18:22:55.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.892+0000 7f8cd432b640 1 -- start start 2024-03-25T18:22:55.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.892+0000 7f8cd432b640 10 monclient: init 2024-03-25T18:22:55.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.892+0000 7f8cd432b640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:55.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.892+0000 7f8cd432b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:55.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.892+0000 7f8cd432b640 10 monclient(hunting): picked mon.noname-a con 0x7f8ccc0b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T18:22:55.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.892+0000 7f8cd432b640 10 monclient(hunting): picked mon.noname-b con 0x7f8ccc0b1730 addr v1:172.21.15.63:6790/0 2024-03-25T18:22:55.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.892+0000 7f8cd432b640 10 monclient(hunting): picked mon.noname-c con 0x7f8ccc14be90 addr v1:172.21.15.81:6789/0 2024-03-25T18:22:55.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.892+0000 7f8cd432b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ccc0af940 con 0x7f8ccc0b4d60 2024-03-25T18:22:55.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.892+0000 7f8cd432b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ccc0b0a70 con 0x7f8ccc14be90 2024-03-25T18:22:55.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.892+0000 7f8cd432b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ccc0b0d50 con 0x7f8ccc0b1730 2024-03-25T18:22:55.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.892+0000 7f8cd432b640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:55.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.892+0000 7f8cd432b640 10 monclient(hunting): authenticate will time out at 8070.079125s 2024-03-25T18:22:55.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.905+0000 7f8cd20c7640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f8ccc0b4d60 0x7f8ccc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:35976/0 (socket says 172.21.15.63:35976) 2024-03-25T18:22:55.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.905+0000 7f8cd20c7640 1 -- 172.21.15.63:0/1840813514 learned_addr learned my addr 172.21.15.63:0/1840813514 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:22:55.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.919+0000 7f8cd10c5640 1 -- 172.21.15.63:0/1840813514 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759409000 0 0) 0x7f8ccc0af940 con 0x7f8ccc0b4d60 2024-03-25T18:22:55.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.919+0000 7f8cd10c5640 10 monclient(hunting): _negotiate 2024-03-25T18:22:55.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.920+0000 7f8cd10c5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:55.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.920+0000 7f8cd10c5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:55.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.920+0000 7f8cd10c5640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:22:55.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.920+0000 7f8cd10c5640 1 -- 172.21.15.63:0/1840813514 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb8003520 con 0x7f8ccc0b4d60 2024-03-25T18:22:55.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd10c5640 1 -- 172.21.15.63:0/1840813514 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1320947535 0 0) 0x7f8cb8003520 con 0x7f8ccc0b4d60 2024-03-25T18:22:55.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd10c5640 1 -- 172.21.15.63:0/1840813514 >> v1:172.21.15.63:6790/0 conn(0x7f8ccc0b1730 legacy=0x7f8ccc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:22:55.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd10c5640 1 -- 172.21.15.63:0/1840813514 >> v1:172.21.15.81:6789/0 conn(0x7f8ccc14be90 legacy=0x7f8ccc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:22:55.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd10c5640 10 monclient: _finish_hunting 0 2024-03-25T18:22:55.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd10c5640 1 monclient: found mon.noname-a 2024-03-25T18:22:55.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd10c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:55.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd10c5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T18:22:55.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd10c5640 1 -- 172.21.15.63:0/1840813514 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ccc0b1030 con 0x7f8ccc0b4d60 2024-03-25T18:22:55.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd10c5640 10 monclient: _finish_auth 0 2024-03-25T18:22:55.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd10c5640 10 monclient: _check_auth_tickets 2024-03-25T18:22:55.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd10c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:55.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd10c5640 1 -- 172.21.15.63:0/1840813514 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8cc8001690 con 0x7f8ccc0b4d60 2024-03-25T18:22:55.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd432b640 5 monclient: authenticate success, global_id 4662 2024-03-25T18:22:55.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd432b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:55.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd10c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:55.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd10c5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:55.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd10c5640 10 monclient: dump: 2024-03-25T18:22:55.940 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:55.940 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:55.940 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:55.941 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:55.941 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:55.941 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:55.941 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:55.941 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:55.941 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:55.941 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:55.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.938+0000 7f8cd432b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:55.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.976+0000 7f8cd10c5640 1 -- 172.21.15.63:0/1840813514 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cc8001860 con 0x7f8ccc0b4d60 2024-03-25T18:22:55.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.976+0000 7f8cd10c5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:55.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.976+0000 7f8cd10c5640 1 -- 172.21.15.63:0/1840813514 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8cc8001cb0 con 0x7f8ccc0b4d60 2024-03-25T18:22:55.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.976+0000 7f8cd10c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:55.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.976+0000 7f8cd10c5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:55.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.976+0000 7f8cd10c5640 10 monclient: dump: 2024-03-25T18:22:55.978 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:55.978 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:55.978 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:55.978 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:55.978 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:55.978 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:55.978 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:55.978 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:55.978 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:55.978 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:55.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.976+0000 7f8cd432b640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:55.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.976+0000 7f8cd432b640 4 set_mon_vals no callback set 2024-03-25T18:22:55.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.977+0000 7f8cd432b640 10 monclient: shutdown 2024-03-25T18:22:55.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.977+0000 7f8cd432b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:55.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.977+0000 7f8cd432b640 1 -- 172.21.15.63:0/1840813514 >> v1:172.21.15.63:6789/0 conn(0x7f8ccc0b4d60 legacy=0x7f8ccc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:55.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.977+0000 7f8cd432b640 1 -- 172.21.15.63:0/1840813514 shutdown_connections 2024-03-25T18:22:55.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.977+0000 7f8cd432b640 1 -- 172.21.15.63:0/1840813514 >> 172.21.15.63:0/1840813514 conn(0x7f8ccc0617a0 msgr2=0x7f8ccc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:55.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.977+0000 7f8cd432b640 1 -- 172.21.15.63:0/1840813514 shutdown_connections 2024-03-25T18:22:55.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.977+0000 7f8cd432b640 1 -- 172.21.15.63:0/1840813514 wait complete. 2024-03-25T18:22:55.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.978+0000 7f8cd432b640 10 monclient: build_initial_monmap 2024-03-25T18:22:55.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.978+0000 7f8cd432b640 10 monclient: monmap: 2024-03-25T18:22:55.981 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:22:55.981 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:55.981 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:22:55.981 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:22:55.981 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:55.981 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:22:55.981 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:22:55.981 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:22:55.982 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:22:55.982 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:55.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.978+0000 7f8cd432b640 1 librados: starting msgr at 2024-03-25T18:22:55.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.978+0000 7f8cd432b640 1 librados: starting objecter 2024-03-25T18:22:55.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.978+0000 7f8cd432b640 1 Processor -- start 2024-03-25T18:22:55.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.978+0000 7f8cd432b640 1 -- start start 2024-03-25T18:22:55.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.978+0000 7f8cd432b640 1 librados: setting wanted keys 2024-03-25T18:22:55.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.978+0000 7f8cd432b640 1 librados: calling monclient init 2024-03-25T18:22:55.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.978+0000 7f8cd432b640 10 monclient: init 2024-03-25T18:22:55.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.979+0000 7f8cd432b640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:55.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.979+0000 7f8cd432b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:55.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.979+0000 7f8cd432b640 10 monclient(hunting): picked mon.noname-a con 0x7f8ccc0b1730 addr v1:172.21.15.63:6789/0 2024-03-25T18:22:55.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.979+0000 7f8cd432b640 10 monclient(hunting): picked mon.noname-b con 0x7f8ccc0b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T18:22:55.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.979+0000 7f8cd432b640 10 monclient(hunting): picked mon.noname-c con 0x7f8ccc14be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:22:55.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.979+0000 7f8cd432b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ccc13baa0 con 0x7f8ccc0b1730 2024-03-25T18:22:55.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.979+0000 7f8cd432b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ccc139440 con 0x7f8ccc0b4d60 2024-03-25T18:22:55.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.979+0000 7f8cd432b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ccc136700 con 0x7f8ccc14be90 2024-03-25T18:22:55.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.979+0000 7f8cd432b640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:55.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.979+0000 7f8cd432b640 10 monclient(hunting): authenticate will time out at 8070.165520s 2024-03-25T18:22:55.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.982+0000 7f8cd28c8640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f8ccc14be90 0x7f8ccc1e2d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:42234/0 (socket says 172.21.15.63:42234) 2024-03-25T18:22:55.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.982+0000 7f8cd28c8640 1 -- 172.21.15.63:0/123918520 learned_addr learned my addr 172.21.15.63:0/123918520 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:22:55.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.983+0000 7f8cc2ffd640 10 client.?.objecter ms_handle_connect 0x7f8ccc14be90 2024-03-25T18:22:55.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.983+0000 7f8cc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:55.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.983+0000 7f8cc2ffd640 10 client.?.objecter ms_handle_connect 0x7f8ccc0b4d60 2024-03-25T18:22:55.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:55.983+0000 7f8cc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:56.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.036+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143214447 0 0) 0x7f8ccc136700 con 0x7f8ccc14be90 2024-03-25T18:22:56.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.036+0000 7f8cc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:56.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.036+0000 7f8cc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:56.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.036+0000 7f8cc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:56.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.036+0000 7f8cc2ffd640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:22:56.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.036+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb0004eb0 con 0x7f8ccc14be90 2024-03-25T18:22:56.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.037+0000 7f8cc2ffd640 10 client.?.objecter ms_handle_connect 0x7f8ccc0b1730 2024-03-25T18:22:56.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.037+0000 7f8cc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:56.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.043+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072746140 0 0) 0x7f8ccc139440 con 0x7f8ccc0b4d60 2024-03-25T18:22:56.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.043+0000 7f8cc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:56.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.043+0000 7f8cc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:56.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.043+0000 7f8cc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:56.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.043+0000 7f8cc2ffd640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:22:56.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.043+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ccc136700 con 0x7f8ccc0b4d60 2024-03-25T18:22:56.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.044+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4225619705 0 0) 0x7f8cb0004eb0 con 0x7f8ccc14be90 2024-03-25T18:22:56.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.044+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ccc139440 con 0x7f8ccc14be90 2024-03-25T18:22:56.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.044+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8cc4001ad0 con 0x7f8ccc14be90 2024-03-25T18:22:56.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.044+0000 7f8cc2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:56.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.044+0000 7f8cc2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:56.106 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.044+0000 7f8cc2ffd640 10 monclient(hunting): dump: 2024-03-25T18:22:56.106 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:56.106 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:56.106 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:56.106 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:56.106 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:56.106 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:56.107 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:56.107 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:56.107 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:56.107 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:56.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.044+0000 7f8cc2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:56.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.057+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975590108 0 0) 0x7f8ccc13baa0 con 0x7f8ccc0b1730 2024-03-25T18:22:56.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.057+0000 7f8cc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:56.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.057+0000 7f8cc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:56.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.057+0000 7f8cc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:56.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.057+0000 7f8cc2ffd640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:22:56.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.057+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb0004eb0 con 0x7f8ccc0b1730 2024-03-25T18:22:56.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.074+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 539354462 0 0) 0x7f8ccc136700 con 0x7f8ccc0b4d60 2024-03-25T18:22:56.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.074+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ccc13baa0 con 0x7f8ccc0b4d60 2024-03-25T18:22:56.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.074+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8cbc001810 con 0x7f8ccc0b4d60 2024-03-25T18:22:56.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.074+0000 7f8cc2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:56.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.074+0000 7f8cc2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:56.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.074+0000 7f8cc2ffd640 10 monclient(hunting): dump: 2024-03-25T18:22:56.109 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:56.109 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:56.109 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:56.109 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:56.109 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:56.109 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:56.110 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:56.110 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:56.110 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:56.110 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:56.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.075+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3285680053 0 0) 0x7f8ccc139440 con 0x7f8ccc14be90 2024-03-25T18:22:56.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.075+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 >> v1:172.21.15.81:6789/0 conn(0x7f8ccc0b4d60 legacy=0x7f8ccc132db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:56.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.075+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 >> v1:172.21.15.63:6789/0 conn(0x7f8ccc0b1730 legacy=0x7f8ccc1d7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:56.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.075+0000 7f8cc2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:56.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.075+0000 7f8cc2ffd640 1 monclient: found mon.c 2024-03-25T18:22:56.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.075+0000 7f8cc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:56.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.075+0000 7f8cc2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:22:56.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.075+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ccc136890 con 0x7f8ccc14be90 2024-03-25T18:22:56.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.075+0000 7f8cc2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:22:56.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.075+0000 7f8cc2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:56.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.075+0000 7f8cc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:56.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.076+0000 7f8cd432b640 5 monclient: authenticate success, global_id 4679 2024-03-25T18:22:56.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.076+0000 7f8cd432b640 10 monclient: _renew_subs 2024-03-25T18:22:56.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.076+0000 7f8cd432b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:22:56.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.076+0000 7f8cd432b640 1 -- 172.21.15.63:0/123918520 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ccc13bec0 con 0x7f8ccc14be90 2024-03-25T18:22:56.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.076+0000 7f8cd432b640 10 client.4679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:56.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.076+0000 7f8cd432b640 10 monclient: _renew_subs 2024-03-25T18:22:56.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.076+0000 7f8cd432b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:22:56.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.076+0000 7f8cd432b640 1 -- 172.21.15.63:0/123918520 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ccc1d8390 con 0x7f8ccc14be90 2024-03-25T18:22:56.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.076+0000 7f8cd432b640 1 librados: init done 2024-03-25T18:22:56.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.077+0000 7f8cd432b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:56.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.077+0000 7f8cd432b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:56.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.077+0000 7f8cd432b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:22:56.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.077+0000 7f8cd432b640 1 -- 172.21.15.63:0/123918520 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ccc1401f0 con 0x7f8ccc14be90 2024-03-25T18:22:56.445 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.443+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cc4001fe0 con 0x7f8ccc14be90 2024-03-25T18:22:56.445 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.443+0000 7f8cc2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:56.445 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.443+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f8cc4004d80 con 0x7f8ccc14be90 2024-03-25T18:22:56.445 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.443+0000 7f8cc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:56.445 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.443+0000 7f8cc2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:56.445 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.443+0000 7f8cc2ffd640 10 monclient: dump: 2024-03-25T18:22:56.445 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:22:56.445 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:22:56.445 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:56.445 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:22:56.445 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:56.445 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:22:56.445 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:22:56.445 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:22:56.446 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:22:56.446 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:22:56.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.443+0000 7f8cd08c4640 4 set_mon_vals no callback set 2024-03-25T18:22:56.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.444+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f8cc401ad50 con 0x7f8ccc14be90 2024-03-25T18:22:56.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.445+0000 7f8cd20c7640 10 monclient: get_auth_request con 0x7f8cb006df70 auth_method 0 2024-03-25T18:22:56.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.445+0000 7f8cc2ffd640 10 client.4679.objecter ms_handle_connect 0x7f8cb006df70 2024-03-25T18:22:56.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.450+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2076230003 0 0) 0x7f8cc4054950 con 0x7f8ccc14be90 2024-03-25T18:22:56.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.450+0000 7f8cc2ffd640 10 client.4679.objecter ms_dispatch 0x7f8ccc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:22:56.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.450+0000 7f8cc2ffd640 3 client.4679.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:22:56.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.450+0000 7f8cc2ffd640 3 client.4679.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:22:56.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.450+0000 7f8cc2ffd640 20 client.4679.objecter dump_active .. 0 homeless 2024-03-25T18:22:56.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.451+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8cc4050290 con 0x7f8ccc14be90 2024-03-25T18:22:56.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.451+0000 7f8cc2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:56.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.451+0000 7f8cc2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:56.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.712+0000 7f8cd432b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:56.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.712+0000 7f8cd432b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:56.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.712+0000 7f8cd432b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:22:56.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.712+0000 7f8cd432b640 1 -- 172.21.15.63:0/123918520 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ccc0b22c0 con 0x7f8ccc14be90 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.748+0000 7f8cc2ffd640 1 -- 172.21.15.63:0/123918520 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 515194931) 0x7f8cc4052390 con 0x7f8ccc14be90 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.748+0000 7f8cc2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.748+0000 7f8cc2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":11,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:22:39.912010+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:22:39.912010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:56.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.750+0000 7f8cd432b640 10 librados: watch_flush enter 2024-03-25T18:22:56.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.750+0000 7f8cd432b640 10 librados: watch_flush exit 2024-03-25T18:22:56.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.750+0000 7f8cd432b640 20 client.4679.objecter shutdown clearing up homeless session... 2024-03-25T18:22:56.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.750+0000 7f8cd432b640 10 client.4679.objecter successfully canceled tick 2024-03-25T18:22:56.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.750+0000 7f8cd432b640 1 -- 172.21.15.63:0/123918520 >> v1:172.21.15.81:6800/3640340873 conn(0x7f8cb006df70 legacy=0x7f8cb0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:56.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.750+0000 7f8cd432b640 10 monclient: shutdown 2024-03-25T18:22:56.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.750+0000 7f8cd432b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:56.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.750+0000 7f8cd432b640 1 -- 172.21.15.63:0/123918520 >> v1:172.21.15.63:6790/0 conn(0x7f8ccc14be90 legacy=0x7f8ccc1e2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:56.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.750+0000 7f8cd432b640 1 -- 172.21.15.63:0/123918520 shutdown_connections 2024-03-25T18:22:56.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.750+0000 7f8cd432b640 1 -- 172.21.15.63:0/123918520 >> 172.21.15.63:0/123918520 conn(0x7f8ccc0617a0 msgr2=0x7f8ccc0b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:56.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.751+0000 7f8cd432b640 1 -- 172.21.15.63:0/123918520 shutdown_connections 2024-03-25T18:22:56.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.751+0000 7f8cd432b640 1 -- 172.21.15.63:0/123918520 wait complete. 2024-03-25T18:22:56.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:22:56.751+0000 7f8cd432b640 1 librados: shutdown 2024-03-25T18:22:56.764 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:03.570 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:03.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.686+0000 7f64d7493640 10 monclient: get_monmap_and_config 2024-03-25T18:23:03.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.686+0000 7f64d7493640 10 monclient: build_initial_monmap 2024-03-25T18:23:03.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.687+0000 7f64d7493640 10 monclient: monmap: 2024-03-25T18:23:03.688 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:23:03.688 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:03.688 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:23:03.688118+0000 2024-03-25T18:23:03.688 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:23:03.688118+0000 2024-03-25T18:23:03.688 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:03.689 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:03.689 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:23:03.689 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:23:03.689 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:23:03.689 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:03.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.687+0000 7f64d7493640 1 Processor -- start 2024-03-25T18:23:03.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.688+0000 7f64d7493640 1 -- start start 2024-03-25T18:23:03.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.688+0000 7f64d7493640 10 monclient: init 2024-03-25T18:23:03.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.688+0000 7f64d7493640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:03.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.688+0000 7f64d7493640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:03.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.688+0000 7f64d7493640 10 monclient(hunting): picked mon.noname-a con 0x7f64d00a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:23:03.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.688+0000 7f64d7493640 10 monclient(hunting): picked mon.noname-c con 0x7f64d013f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:23:03.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.688+0000 7f64d7493640 10 monclient(hunting): picked mon.noname-b con 0x7f64d01430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:23:03.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.688+0000 7f64d7493640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d0142520 con 0x7f64d00a98f0 2024-03-25T18:23:03.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.688+0000 7f64d7493640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d00b28a0 con 0x7f64d013f710 2024-03-25T18:23:03.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.688+0000 7f64d7493640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d00b2a30 con 0x7f64d01430c0 2024-03-25T18:23:03.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.688+0000 7f64d7493640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:03.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.688+0000 7f64d7493640 10 monclient(hunting): authenticate will time out at 8077.875120s 2024-03-25T18:23:03.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.703+0000 7f64d4a2e640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f64d013f710 0x7f64d01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:50060/0 (socket says 172.21.15.63:50060) 2024-03-25T18:23:03.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.703+0000 7f64d4a2e640 1 -- 172.21.15.63:0/1487535420 learned_addr learned my addr 172.21.15.63:0/1487535420 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:23:03.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.711+0000 7f64c7fff640 1 -- 172.21.15.63:0/1487535420 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001113920 0 0) 0x7f64d0142520 con 0x7f64d00a98f0 2024-03-25T18:23:03.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.711+0000 7f64c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:03.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.711+0000 7f64c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:03.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.711+0000 7f64c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:03.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.711+0000 7f64c7fff640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:23:03.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.711+0000 7f64c7fff640 1 -- 172.21.15.63:0/1487535420 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64bc003520 con 0x7f64d00a98f0 2024-03-25T18:23:03.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.713+0000 7f64c7fff640 1 -- 172.21.15.63:0/1487535420 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596033877 0 0) 0x7f64d00b2a30 con 0x7f64d01430c0 2024-03-25T18:23:03.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.713+0000 7f64c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:03.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.713+0000 7f64c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:03.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.713+0000 7f64c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:03.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.713+0000 7f64c7fff640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:23:03.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.713+0000 7f64c7fff640 1 -- 172.21.15.63:0/1487535420 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64d0142520 con 0x7f64d01430c0 2024-03-25T18:23:03.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.713+0000 7f64c7fff640 1 -- 172.21.15.63:0/1487535420 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879480222 0 0) 0x7f64d00b28a0 con 0x7f64d013f710 2024-03-25T18:23:03.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.713+0000 7f64c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:03.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.713+0000 7f64c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:03.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.713+0000 7f64c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:03.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.713+0000 7f64c7fff640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:23:03.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.714+0000 7f64c7fff640 1 -- 172.21.15.63:0/1487535420 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64d00b2a30 con 0x7f64d013f710 2024-03-25T18:23:03.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.721+0000 7f64c7fff640 1 -- 172.21.15.63:0/1487535420 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2755602735 0 0) 0x7f64d0142520 con 0x7f64d01430c0 2024-03-25T18:23:03.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.721+0000 7f64c7fff640 1 -- 172.21.15.63:0/1487535420 >> v1:172.21.15.81:6789/0 conn(0x7f64d013f710 legacy=0x7f64d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:03.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.721+0000 7f64c7fff640 1 -- 172.21.15.63:0/1487535420 >> v1:172.21.15.63:6789/0 conn(0x7f64d00a98f0 legacy=0x7f64d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:03.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.721+0000 7f64c7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:23:03.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.721+0000 7f64c7fff640 1 monclient: found mon.noname-b 2024-03-25T18:23:03.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.721+0000 7f64c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:03.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.721+0000 7f64c7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:23:03.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.721+0000 7f64c7fff640 1 -- 172.21.15.63:0/1487535420 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d00b44c0 con 0x7f64d01430c0 2024-03-25T18:23:03.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.721+0000 7f64c7fff640 10 monclient: _finish_auth 0 2024-03-25T18:23:03.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.721+0000 7f64c7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:23:03.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.721+0000 7f64c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:03.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.721+0000 7f64d7493640 5 monclient: authenticate success, global_id 4682 2024-03-25T18:23:03.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.721+0000 7f64d7493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:03.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.723+0000 7f64c7fff640 1 -- 172.21.15.63:0/1487535420 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f64c80019b0 con 0x7f64d01430c0 2024-03-25T18:23:03.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.723+0000 7f64c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:03.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.723+0000 7f64c7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:03.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.723+0000 7f64c7fff640 10 monclient: dump: 2024-03-25T18:23:03.724 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:03.724 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:03.724 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:03.725 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:03.725 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:03.725 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:03.725 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:03.725 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:03.725 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:03.725 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:03.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.723+0000 7f64d7493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:03.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.730+0000 7f64c7fff640 1 -- 172.21.15.63:0/1487535420 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64c8001c00 con 0x7f64d01430c0 2024-03-25T18:23:03.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.730+0000 7f64c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:03.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.730+0000 7f64c7fff640 1 -- 172.21.15.63:0/1487535420 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f64c8001f90 con 0x7f64d01430c0 2024-03-25T18:23:03.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.730+0000 7f64c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:03.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.730+0000 7f64c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:03.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.730+0000 7f64c7fff640 10 monclient: dump: 2024-03-25T18:23:03.732 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:03.732 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:03.732 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:03.732 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:03.732 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:03.733 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:03.733 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:03.733 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:03.733 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:03.733 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:03.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.730+0000 7f64d7493640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:03.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.730+0000 7f64d7493640 4 set_mon_vals no callback set 2024-03-25T18:23:03.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.731+0000 7f64d7493640 10 monclient: shutdown 2024-03-25T18:23:03.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.731+0000 7f64d7493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:03.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.731+0000 7f64d7493640 1 -- 172.21.15.63:0/1487535420 >> v1:172.21.15.63:6790/0 conn(0x7f64d01430c0 legacy=0x7f64d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:03.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.731+0000 7f64d7493640 1 -- 172.21.15.63:0/1487535420 shutdown_connections 2024-03-25T18:23:03.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.731+0000 7f64d7493640 1 -- 172.21.15.63:0/1487535420 >> 172.21.15.63:0/1487535420 conn(0x7f64d00b7ea0 msgr2=0x7f64d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:03.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.731+0000 7f64d7493640 1 -- 172.21.15.63:0/1487535420 shutdown_connections 2024-03-25T18:23:03.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.731+0000 7f64d7493640 1 -- 172.21.15.63:0/1487535420 wait complete. 2024-03-25T18:23:03.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.732+0000 7f64d7493640 10 monclient: build_initial_monmap 2024-03-25T18:23:03.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.732+0000 7f64d7493640 10 monclient: monmap: 2024-03-25T18:23:03.736 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:23:03.736 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:03.736 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:23:03.737 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:23:03.737 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:03.737 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:23:03.737 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:23:03.737 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:23:03.737 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:23:03.737 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:03.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.733+0000 7f64d7493640 1 librados: starting msgr at 2024-03-25T18:23:03.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.733+0000 7f64d7493640 1 librados: starting objecter 2024-03-25T18:23:03.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.733+0000 7f64d7493640 1 Processor -- start 2024-03-25T18:23:03.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.733+0000 7f64d7493640 1 -- start start 2024-03-25T18:23:03.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.733+0000 7f64d7493640 1 librados: setting wanted keys 2024-03-25T18:23:03.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.733+0000 7f64d7493640 1 librados: calling monclient init 2024-03-25T18:23:03.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.733+0000 7f64d7493640 10 monclient: init 2024-03-25T18:23:03.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.734+0000 7f64d7493640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:03.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.734+0000 7f64d7493640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:03.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.734+0000 7f64d7493640 10 monclient(hunting): picked mon.noname-c con 0x7f64d00a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:23:03.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.734+0000 7f64d7493640 10 monclient(hunting): picked mon.noname-b con 0x7f64d013f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:23:03.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.734+0000 7f64d7493640 10 monclient(hunting): picked mon.noname-a con 0x7f64d01430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:23:03.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.734+0000 7f64d7493640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d013b820 con 0x7f64d01430c0 2024-03-25T18:23:03.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.734+0000 7f64d7493640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d00b7c20 con 0x7f64d013f710 2024-03-25T18:23:03.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.734+0000 7f64d7493640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d013ba90 con 0x7f64d00a98f0 2024-03-25T18:23:03.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.734+0000 7f64d7493640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:03.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.734+0000 7f64d7493640 10 monclient(hunting): authenticate will time out at 8077.920722s 2024-03-25T18:23:03.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.748+0000 7f64d522f640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f64d00a98f0 0x7f64d013beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:42718/0 (socket says 172.21.15.63:42718) 2024-03-25T18:23:03.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.748+0000 7f64d522f640 1 -- 172.21.15.63:0/1705989849 learned_addr learned my addr 172.21.15.63:0/1705989849 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:23:03.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.751+0000 7f64c5ffb640 10 client.?.objecter ms_handle_connect 0x7f64d00a98f0 2024-03-25T18:23:03.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.751+0000 7f64c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:03.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.751+0000 7f64c5ffb640 10 client.?.objecter ms_handle_connect 0x7f64d013f710 2024-03-25T18:23:03.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.751+0000 7f64c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:03.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.753+0000 7f64c5ffb640 10 client.?.objecter ms_handle_connect 0x7f64d01430c0 2024-03-25T18:23:03.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.753+0000 7f64c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:03.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.758+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467647274 0 0) 0x7f64d013ba90 con 0x7f64d00a98f0 2024-03-25T18:23:03.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.758+0000 7f64c5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:03.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.758+0000 7f64c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:03.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.758+0000 7f64c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:03.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.758+0000 7f64c5ffb640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:23:03.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.758+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b4004eb0 con 0x7f64d00a98f0 2024-03-25T18:23:03.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.764+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865717173 0 0) 0x7f64d013b820 con 0x7f64d01430c0 2024-03-25T18:23:03.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.764+0000 7f64c5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:03.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.764+0000 7f64c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:03.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.764+0000 7f64c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:03.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.764+0000 7f64c5ffb640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:23:03.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.764+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64d013ba90 con 0x7f64d01430c0 2024-03-25T18:23:03.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.765+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117005721 0 0) 0x7f64d00b7c20 con 0x7f64d013f710 2024-03-25T18:23:03.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.765+0000 7f64c5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:03.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.765+0000 7f64c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:03.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.765+0000 7f64c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:03.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.765+0000 7f64c5ffb640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:23:03.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.765+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64d013b820 con 0x7f64d013f710 2024-03-25T18:23:03.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.783+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2516179488 0 0) 0x7f64b4004eb0 con 0x7f64d00a98f0 2024-03-25T18:23:03.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.783+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64d00b7c20 con 0x7f64d00a98f0 2024-03-25T18:23:03.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.783+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f64cc001990 con 0x7f64d00a98f0 2024-03-25T18:23:03.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.783+0000 7f64c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:03.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.783+0000 7f64c5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:03.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.783+0000 7f64c5ffb640 10 monclient(hunting): dump: 2024-03-25T18:23:03.785 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:03.785 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:03.785 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:03.785 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:03.785 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:03.785 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:03.785 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:03.785 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:03.785 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:03.785 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:03.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.783+0000 7f64c5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:03.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.786+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2152141097 0 0) 0x7f64d013ba90 con 0x7f64d01430c0 2024-03-25T18:23:03.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.786+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64b4004eb0 con 0x7f64d01430c0 2024-03-25T18:23:03.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.786+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f64c8001f10 con 0x7f64d01430c0 2024-03-25T18:23:03.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.786+0000 7f64c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:03.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.786+0000 7f64c5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:03.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.786+0000 7f64c5ffb640 10 monclient(hunting): dump: 2024-03-25T18:23:03.792 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:03.792 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:03.792 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:03.792 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:03.792 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:03.792 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:03.792 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:03.792 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:03.792 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:03.792 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:03.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.789+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3604123873 0 0) 0x7f64d013b820 con 0x7f64d013f710 2024-03-25T18:23:03.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.789+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64d013ba90 con 0x7f64d013f710 2024-03-25T18:23:03.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.789+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f64c0001790 con 0x7f64d013f710 2024-03-25T18:23:03.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.789+0000 7f64c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:03.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.789+0000 7f64c5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:03.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.789+0000 7f64c5ffb640 10 monclient(hunting): dump: 2024-03-25T18:23:03.793 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:03.793 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:03.793 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:03.793 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:03.793 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:03.793 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:03.793 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:03.793 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:03.793 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:03.793 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:03.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.799+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2321880877 0 0) 0x7f64b4004eb0 con 0x7f64d01430c0 2024-03-25T18:23:03.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.799+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 >> v1:172.21.15.63:6790/0 conn(0x7f64d00a98f0 legacy=0x7f64d013beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:03.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.799+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 >> v1:172.21.15.81:6789/0 conn(0x7f64d013f710 legacy=0x7f64d00b5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:03.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.799+0000 7f64c5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:23:03.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.799+0000 7f64c5ffb640 1 monclient: found mon.a 2024-03-25T18:23:03.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.799+0000 7f64c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:03.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.799+0000 7f64c5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:03.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.799+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d0139d20 con 0x7f64d01430c0 2024-03-25T18:23:03.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.799+0000 7f64c5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:23:03.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.799+0000 7f64c5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:23:03.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.799+0000 7f64c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:03.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.799+0000 7f64d7493640 5 monclient: authenticate success, global_id 4671 2024-03-25T18:23:03.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.799+0000 7f64d7493640 10 monclient: _renew_subs 2024-03-25T18:23:03.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.799+0000 7f64d7493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:03.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.799+0000 7f64d7493640 1 -- 172.21.15.63:0/1705989849 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d0139f90 con 0x7f64d01430c0 2024-03-25T18:23:03.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.800+0000 7f64d7493640 10 client.4671.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:03.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.800+0000 7f64d7493640 10 monclient: _renew_subs 2024-03-25T18:23:03.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.800+0000 7f64d7493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:03.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.800+0000 7f64d7493640 1 -- 172.21.15.63:0/1705989849 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d00b68b0 con 0x7f64d01430c0 2024-03-25T18:23:03.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.800+0000 7f64d7493640 1 librados: init done 2024-03-25T18:23:03.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.801+0000 7f64d7493640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:03.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.801+0000 7f64d7493640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:03.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.801+0000 7f64d7493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:03.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:03.801+0000 7f64d7493640 1 -- 172.21.15.63:0/1705989849 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d00b3eb0 con 0x7f64d01430c0 2024-03-25T18:23:04.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.110+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64c8003c20 con 0x7f64d01430c0 2024-03-25T18:23:04.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.110+0000 7f64c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:04.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.110+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f64c8003f30 con 0x7f64d01430c0 2024-03-25T18:23:04.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.110+0000 7f64c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:04.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.111+0000 7f64c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:04.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.111+0000 7f64c5ffb640 10 monclient: dump: 2024-03-25T18:23:04.112 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:04.112 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:04.112 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:04.112 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:04.113 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:04.113 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:04.113 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:04.113 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:04.113 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:04.113 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:04.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.110+0000 7f64c77fe640 4 set_mon_vals no callback set 2024-03-25T18:23:04.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.111+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f64c8019f00 con 0x7f64d01430c0 2024-03-25T18:23:04.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.112+0000 7f64d522f640 10 monclient: get_auth_request con 0x7f64b406e0e0 auth_method 0 2024-03-25T18:23:04.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.112+0000 7f64c5ffb640 10 client.4671.objecter ms_handle_connect 0x7f64b406e0e0 2024-03-25T18:23:04.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.117+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2076230003 0 0) 0x7f64c80539b0 con 0x7f64d01430c0 2024-03-25T18:23:04.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.117+0000 7f64c5ffb640 10 client.4671.objecter ms_dispatch 0x7f64d0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:23:04.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.117+0000 7f64c5ffb640 3 client.4671.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:23:04.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.117+0000 7f64c5ffb640 3 client.4671.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:23:04.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.117+0000 7f64c5ffb640 20 client.4671.objecter dump_active .. 0 homeless 2024-03-25T18:23:04.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.117+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64c804f4c0 con 0x7f64d01430c0 2024-03-25T18:23:04.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.117+0000 7f64c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:04.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.117+0000 7f64c5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:04.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.376+0000 7f64d7493640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:04.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.376+0000 7f64d7493640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:04.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.376+0000 7f64d7493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:04.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.376+0000 7f64d7493640 1 -- 172.21.15.63:0/1705989849 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64d013a120 con 0x7f64d01430c0 2024-03-25T18:23:04.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.411+0000 7f64c5ffb640 1 -- 172.21.15.63:0/1705989849 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 515194931) 0x7f64c80513f0 con 0x7f64d01430c0 2024-03-25T18:23:04.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.411+0000 7f64c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:04.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.411+0000 7f64c5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:04.413 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:23:04.413 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":11,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:22:39.912010+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:22:39.912010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:04.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.414+0000 7f64d7493640 10 librados: watch_flush enter 2024-03-25T18:23:04.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.414+0000 7f64d7493640 10 librados: watch_flush exit 2024-03-25T18:23:04.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.414+0000 7f64d7493640 20 client.4671.objecter shutdown clearing up homeless session... 2024-03-25T18:23:04.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.414+0000 7f64d7493640 10 client.4671.objecter successfully canceled tick 2024-03-25T18:23:04.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.415+0000 7f64d7493640 1 -- 172.21.15.63:0/1705989849 >> v1:172.21.15.81:6800/3640340873 conn(0x7f64b406e0e0 legacy=0x7f64b40705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:04.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.415+0000 7f64d7493640 10 monclient: shutdown 2024-03-25T18:23:04.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.415+0000 7f64d7493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:04.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.415+0000 7f64d7493640 1 -- 172.21.15.63:0/1705989849 >> v1:172.21.15.63:6789/0 conn(0x7f64d01430c0 legacy=0x7f64d00b61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:04.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.415+0000 7f64d7493640 1 -- 172.21.15.63:0/1705989849 shutdown_connections 2024-03-25T18:23:04.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.415+0000 7f64d7493640 1 -- 172.21.15.63:0/1705989849 >> 172.21.15.63:0/1705989849 conn(0x7f64d00b7ea0 msgr2=0x7f64d0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:04.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.415+0000 7f64d7493640 1 -- 172.21.15.63:0/1705989849 shutdown_connections 2024-03-25T18:23:04.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.415+0000 7f64d7493640 1 -- 172.21.15.63:0/1705989849 wait complete. 2024-03-25T18:23:04.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:04.415+0000 7f64d7493640 1 librados: shutdown 2024-03-25T18:23:04.427 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:05.057 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:46.749 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:05.105 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:46.797 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:05.145 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:46.838 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:05.164 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:10:46.824 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:05.173 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:10:46.833 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:05.176 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:46.869 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:05.187 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:46.879 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:05.217 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:10:46.877 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:05.229 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:46.921 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:11.234 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:11.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.349+0000 7fa5c908d640 10 monclient: get_monmap_and_config 2024-03-25T18:23:11.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.349+0000 7fa5c908d640 10 monclient: build_initial_monmap 2024-03-25T18:23:11.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.349+0000 7fa5c908d640 10 monclient: monmap: 2024-03-25T18:23:11.349 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:23:11.349 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:11.350 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:23:11.349565+0000 2024-03-25T18:23:11.350 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:23:11.349565+0000 2024-03-25T18:23:11.350 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:11.350 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:11.350 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:23:11.350 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:23:11.350 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:23:11.350 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:11.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.350+0000 7fa5c908d640 1 Processor -- start 2024-03-25T18:23:11.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.350+0000 7fa5c908d640 1 -- start start 2024-03-25T18:23:11.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.350+0000 7fa5c908d640 10 monclient: init 2024-03-25T18:23:11.352 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.351+0000 7fa5c908d640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:11.352 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.351+0000 7fa5c908d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:11.352 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.351+0000 7fa5c908d640 10 monclient(hunting): picked mon.noname-a con 0x7fa5c40b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T18:23:11.352 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.351+0000 7fa5c908d640 10 monclient(hunting): picked mon.noname-b con 0x7fa5c40b1730 addr v1:172.21.15.63:6790/0 2024-03-25T18:23:11.352 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.351+0000 7fa5c908d640 10 monclient(hunting): picked mon.noname-c con 0x7fa5c414be90 addr v1:172.21.15.81:6789/0 2024-03-25T18:23:11.352 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.351+0000 7fa5c908d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c40af940 con 0x7fa5c40b4d60 2024-03-25T18:23:11.352 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.351+0000 7fa5c908d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c40b0a70 con 0x7fa5c414be90 2024-03-25T18:23:11.352 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.351+0000 7fa5c908d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c40b0d50 con 0x7fa5c40b1730 2024-03-25T18:23:11.352 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.351+0000 7fa5c908d640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:11.352 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.351+0000 7fa5c908d640 10 monclient(hunting): authenticate will time out at 8085.536811s 2024-03-25T18:23:11.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.355+0000 7fa5c359e640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fa5c414be90 0x7fa5c414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:41478/0 (socket says 172.21.15.63:41478) 2024-03-25T18:23:11.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.355+0000 7fa5c359e640 1 -- 172.21.15.63:0/2744909711 learned_addr learned my addr 172.21.15.63:0/2744909711 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:23:11.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.367+0000 7fa5c1d9b640 1 -- 172.21.15.63:0/2744909711 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741776453 0 0) 0x7fa5c40b0a70 con 0x7fa5c414be90 2024-03-25T18:23:11.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.367+0000 7fa5c1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:23:11.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.367+0000 7fa5c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:11.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.367+0000 7fa5c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:11.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.367+0000 7fa5c1d9b640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:23:11.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.367+0000 7fa5c1d9b640 1 -- 172.21.15.63:0/2744909711 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ac003520 con 0x7fa5c414be90 2024-03-25T18:23:11.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c1d9b640 1 -- 172.21.15.63:0/2744909711 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3412136377 0 0) 0x7fa5ac003520 con 0x7fa5c414be90 2024-03-25T18:23:11.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c1d9b640 1 -- 172.21.15.63:0/2744909711 >> v1:172.21.15.63:6790/0 conn(0x7fa5c40b1730 legacy=0x7fa5c40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:11.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c1d9b640 1 -- 172.21.15.63:0/2744909711 >> v1:172.21.15.63:6789/0 conn(0x7fa5c40b4d60 legacy=0x7fa5c413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:23:11.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:23:11.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c1d9b640 1 monclient: found mon.noname-c 2024-03-25T18:23:11.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:11.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:23:11.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c1d9b640 1 -- 172.21.15.63:0/2744909711 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c40b1030 con 0x7fa5c414be90 2024-03-25T18:23:11.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:23:11.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:23:11.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:11.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c1d9b640 1 -- 172.21.15.63:0/2744909711 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fa5bc001a10 con 0x7fa5c414be90 2024-03-25T18:23:11.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:11.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:11.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c1d9b640 10 monclient: dump: 2024-03-25T18:23:11.378 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:11.378 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:11.378 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:11.378 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:11.378 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:11.378 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:11.378 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:11.378 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:11.378 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:11.378 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:11.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c908d640 5 monclient: authenticate success, global_id 4711 2024-03-25T18:23:11.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.376+0000 7fa5c908d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:11.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.397+0000 7fa5c1d9b640 1 -- 172.21.15.63:0/2744909711 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5bc001be0 con 0x7fa5c414be90 2024-03-25T18:23:11.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.397+0000 7fa5c1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:11.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.397+0000 7fa5c1d9b640 1 -- 172.21.15.63:0/2744909711 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fa5bc002030 con 0x7fa5c414be90 2024-03-25T18:23:11.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.397+0000 7fa5c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:11.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.397+0000 7fa5c1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:11.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.397+0000 7fa5c1d9b640 10 monclient: dump: 2024-03-25T18:23:11.398 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:11.398 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:11.398 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:11.398 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:11.398 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:11.398 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:11.398 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:11.398 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:11.398 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:11.398 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:11.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.397+0000 7fa5c908d640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:11.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.397+0000 7fa5c908d640 4 set_mon_vals no callback set 2024-03-25T18:23:11.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.398+0000 7fa5c908d640 10 monclient: shutdown 2024-03-25T18:23:11.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.398+0000 7fa5c908d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:11.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.398+0000 7fa5c908d640 1 -- 172.21.15.63:0/2744909711 >> v1:172.21.15.81:6789/0 conn(0x7fa5c414be90 legacy=0x7fa5c414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:11.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.398+0000 7fa5c908d640 1 -- 172.21.15.63:0/2744909711 shutdown_connections 2024-03-25T18:23:11.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.398+0000 7fa5c908d640 1 -- 172.21.15.63:0/2744909711 >> 172.21.15.63:0/2744909711 conn(0x7fa5c40617a0 msgr2=0x7fa5c4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:11.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.398+0000 7fa5c908d640 1 -- 172.21.15.63:0/2744909711 shutdown_connections 2024-03-25T18:23:11.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.398+0000 7fa5c908d640 1 -- 172.21.15.63:0/2744909711 wait complete. 2024-03-25T18:23:11.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.399+0000 7fa5c908d640 10 monclient: build_initial_monmap 2024-03-25T18:23:11.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.399+0000 7fa5c908d640 10 monclient: monmap: 2024-03-25T18:23:11.400 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:23:11.400 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:11.400 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:23:11.400 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:23:11.400 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:11.401 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:23:11.401 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:23:11.401 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:23:11.401 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:23:11.401 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:11.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.400+0000 7fa5c908d640 1 librados: starting msgr at 2024-03-25T18:23:11.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.400+0000 7fa5c908d640 1 librados: starting objecter 2024-03-25T18:23:11.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.400+0000 7fa5c908d640 1 Processor -- start 2024-03-25T18:23:11.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.400+0000 7fa5c908d640 1 -- start start 2024-03-25T18:23:11.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.400+0000 7fa5c908d640 1 librados: setting wanted keys 2024-03-25T18:23:11.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.400+0000 7fa5c908d640 1 librados: calling monclient init 2024-03-25T18:23:11.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.400+0000 7fa5c908d640 10 monclient: init 2024-03-25T18:23:11.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.401+0000 7fa5c908d640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:11.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.401+0000 7fa5c908d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:11.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.401+0000 7fa5c908d640 10 monclient(hunting): picked mon.noname-b con 0x7fa5c40b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:23:11.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.401+0000 7fa5c908d640 10 monclient(hunting): picked mon.noname-c con 0x7fa5c40b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T18:23:11.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.401+0000 7fa5c908d640 10 monclient(hunting): picked mon.noname-a con 0x7fa5c414be90 addr v1:172.21.15.63:6789/0 2024-03-25T18:23:11.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.401+0000 7fa5c908d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c41d7a40 con 0x7fa5c414be90 2024-03-25T18:23:11.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.401+0000 7fa5c908d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c413a9d0 con 0x7fa5c40b1730 2024-03-25T18:23:11.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.401+0000 7fa5c908d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c41d85e0 con 0x7fa5c40b4d60 2024-03-25T18:23:11.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.401+0000 7fa5c908d640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:11.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.401+0000 7fa5c908d640 10 monclient(hunting): authenticate will time out at 8085.587247s 2024-03-25T18:23:11.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.407+0000 7fa5c2d9d640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fa5c40b1730 0x7fa5c4139140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:41494/0 (socket says 172.21.15.63:41494) 2024-03-25T18:23:11.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.407+0000 7fa5c2d9d640 1 -- 172.21.15.63:0/2191243613 learned_addr learned my addr 172.21.15.63:0/2191243613 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:23:11.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.413+0000 7fa5ab7fe640 10 client.?.objecter ms_handle_connect 0x7fa5c40b1730 2024-03-25T18:23:11.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.413+0000 7fa5ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:11.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.415+0000 7fa5ab7fe640 10 client.?.objecter ms_handle_connect 0x7fa5c40b4d60 2024-03-25T18:23:11.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.415+0000 7fa5ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:11.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.415+0000 7fa5ab7fe640 10 client.?.objecter ms_handle_connect 0x7fa5c414be90 2024-03-25T18:23:11.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.415+0000 7fa5ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:11.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.418+0000 7fa5c259c640 1 -- 172.21.15.63:0/2191243613 >> v1:172.21.15.63:6790/0 conn(0x7fa5c40b4d60 legacy=0x7fa5c41d7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:23:11.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.418+0000 7fa5c259c640 1 -- 172.21.15.63:0/2191243613 >> v1:172.21.15.63:6790/0 conn(0x7fa5c40b4d60 legacy=0x7fa5c41d7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:23:11.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.418+0000 7fa5c259c640 1 --1- 172.21.15.63:0/2191243613 >> v1:172.21.15.63:6790/0 conn(0x7fa5c40b4d60 0x7fa5c41d7ed0 :-1 s=OPENED pgs=79271 cs=1 l=1).handle_message read tag failed 2024-03-25T18:23:11.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.418+0000 7fa5c259c640 1 --1- 172.21.15.63:0/2191243613 >> v1:172.21.15.63:6790/0 conn(0x7fa5c40b4d60 0x7fa5c41d7ed0 :-1 s=OPENED pgs=79271 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:23:11.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.419+0000 7fa5ab7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.63:6790/0 2024-03-25T18:23:11.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.425+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757525855 0 0) 0x7fa5c41d7a40 con 0x7fa5c414be90 2024-03-25T18:23:11.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.425+0000 7fa5ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:11.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.425+0000 7fa5ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:11.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.425+0000 7fa5ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:11.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.425+0000 7fa5ab7fe640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:23:11.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.425+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b0004eb0 con 0x7fa5c414be90 2024-03-25T18:23:11.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.426+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040971958 0 0) 0x7fa5c413a9d0 con 0x7fa5c40b1730 2024-03-25T18:23:11.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.426+0000 7fa5ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:11.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.426+0000 7fa5ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:11.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.426+0000 7fa5ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:11.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.426+0000 7fa5ab7fe640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:23:11.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.426+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5c41d7a40 con 0x7fa5c40b1730 2024-03-25T18:23:11.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.439+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3895820979 0 0) 0x7fa5b0004eb0 con 0x7fa5c414be90 2024-03-25T18:23:11.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.440+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5c413a9d0 con 0x7fa5c414be90 2024-03-25T18:23:11.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.440+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fa5bc001ef0 con 0x7fa5c414be90 2024-03-25T18:23:11.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.440+0000 7fa5ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:11.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.440+0000 7fa5ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:11.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.440+0000 7fa5ab7fe640 10 monclient(hunting): dump: 2024-03-25T18:23:11.440 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:11.440 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:11.440 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:11.440 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:11.440 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:11.440 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:11.440 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:11.441 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:11.441 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:11.441 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:11.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.440+0000 7fa5ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:11.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3601625281 0 0) 0x7fa5c413a9d0 con 0x7fa5c414be90 2024-03-25T18:23:11.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 >> v1:172.21.15.63:6790/0 conn(0x7fa5c40b4d60 legacy=0x7fa5c41d7ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:23:11.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 >> v1:172.21.15.81:6789/0 conn(0x7fa5c40b1730 legacy=0x7fa5c4139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:11.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:23:11.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5ab7fe640 1 monclient: found mon.a 2024-03-25T18:23:11.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:11.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5ab7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:11.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c413bf30 con 0x7fa5c414be90 2024-03-25T18:23:11.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5ab7fe640 10 monclient: _finish_auth 0 2024-03-25T18:23:11.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:23:11.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:11.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5c908d640 5 monclient: authenticate success, global_id 4674 2024-03-25T18:23:11.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5c908d640 10 monclient: _renew_subs 2024-03-25T18:23:11.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5c908d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:11.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5c908d640 1 -- 172.21.15.63:0/2191243613 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c413c1e0 con 0x7fa5c414be90 2024-03-25T18:23:11.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5c908d640 10 client.4674.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:11.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5c908d640 10 monclient: _renew_subs 2024-03-25T18:23:11.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5c908d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:11.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5c908d640 1 -- 172.21.15.63:0/2191243613 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c4142b90 con 0x7fa5c414be90 2024-03-25T18:23:11.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.446+0000 7fa5c908d640 1 librados: init done 2024-03-25T18:23:11.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.447+0000 7fa5c908d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:11.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.447+0000 7fa5c908d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:11.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.447+0000 7fa5c908d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:11.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.447+0000 7fa5c908d640 1 -- 172.21.15.63:0/2191243613 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c41d77a0 con 0x7fa5c414be90 2024-03-25T18:23:11.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.768+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5bc002090 con 0x7fa5c414be90 2024-03-25T18:23:11.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.768+0000 7fa5ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:11.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.768+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fa5bc004fb0 con 0x7fa5c414be90 2024-03-25T18:23:11.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.768+0000 7fa5ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:11.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.768+0000 7fa5ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:11.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.768+0000 7fa5ab7fe640 10 monclient: dump: 2024-03-25T18:23:11.769 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:11.769 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:11.769 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:11.769 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:11.769 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:11.769 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:11.770 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:11.770 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:11.770 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:11.770 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:11.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.768+0000 7fa5c159a640 4 set_mon_vals no callback set 2024-03-25T18:23:11.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.769+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fa5bc01af80 con 0x7fa5c414be90 2024-03-25T18:23:11.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.770+0000 7fa5c2d9d640 10 monclient: get_auth_request con 0x7fa5b006dcf0 auth_method 0 2024-03-25T18:23:11.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.770+0000 7fa5ab7fe640 10 client.4674.objecter ms_handle_connect 0x7fa5b006dcf0 2024-03-25T18:23:11.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.776+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2076230003 0 0) 0x7fa5bc054a20 con 0x7fa5c414be90 2024-03-25T18:23:11.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.776+0000 7fa5ab7fe640 10 client.4674.objecter ms_dispatch 0x7fa5c4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:23:11.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.776+0000 7fa5ab7fe640 3 client.4674.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:23:11.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.776+0000 7fa5ab7fe640 3 client.4674.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:23:11.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.776+0000 7fa5ab7fe640 20 client.4674.objecter dump_active .. 0 homeless 2024-03-25T18:23:11.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.776+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5bc050530 con 0x7fa5c414be90 2024-03-25T18:23:11.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.776+0000 7fa5ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:11.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:11.776+0000 7fa5ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:12.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.029+0000 7fa5c908d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:12.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.029+0000 7fa5c908d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:12.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.029+0000 7fa5c908d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:12.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.029+0000 7fa5c908d640 1 -- 172.21.15.63:0/2191243613 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5c41d7bf0 con 0x7fa5c414be90 2024-03-25T18:23:12.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.056+0000 7fa5ab7fe640 1 -- 172.21.15.63:0/2191243613 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 515194931) 0x7fa5c41d7bf0 con 0x7fa5c414be90 2024-03-25T18:23:12.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.056+0000 7fa5ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:12.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.056+0000 7fa5ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:12.057 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:23:12.057 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":11,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:22:39.912010+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:22:39.912010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:12.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.058+0000 7fa5c908d640 10 librados: watch_flush enter 2024-03-25T18:23:12.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.058+0000 7fa5c908d640 10 librados: watch_flush exit 2024-03-25T18:23:12.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.058+0000 7fa5c908d640 20 client.4674.objecter shutdown clearing up homeless session... 2024-03-25T18:23:12.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.058+0000 7fa5c908d640 10 client.4674.objecter successfully canceled tick 2024-03-25T18:23:12.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.059+0000 7fa5c908d640 1 -- 172.21.15.63:0/2191243613 >> v1:172.21.15.81:6800/3640340873 conn(0x7fa5b006dcf0 legacy=0x7fa5b00701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.059+0000 7fa5c908d640 10 monclient: shutdown 2024-03-25T18:23:12.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.059+0000 7fa5c908d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:12.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.059+0000 7fa5c908d640 1 -- 172.21.15.63:0/2191243613 >> v1:172.21.15.63:6789/0 conn(0x7fa5c414be90 legacy=0x7fa5c413b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.059+0000 7fa5c908d640 1 -- 172.21.15.63:0/2191243613 shutdown_connections 2024-03-25T18:23:12.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.059+0000 7fa5c908d640 1 -- 172.21.15.63:0/2191243613 >> 172.21.15.63:0/2191243613 conn(0x7fa5c40617a0 msgr2=0x7fa5c4137120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:12.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.059+0000 7fa5c908d640 1 -- 172.21.15.63:0/2191243613 shutdown_connections 2024-03-25T18:23:12.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.059+0000 7fa5c908d640 1 -- 172.21.15.63:0/2191243613 wait complete. 2024-03-25T18:23:12.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:12.059+0000 7fa5c908d640 1 librados: shutdown 2024-03-25T18:23:12.071 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:18.877 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:18.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:18.992+0000 7fe479e51640 10 monclient: get_monmap_and_config 2024-03-25T18:23:18.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:18.992+0000 7fe479e51640 10 monclient: build_initial_monmap 2024-03-25T18:23:18.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:18.993+0000 7fe479e51640 10 monclient: monmap: 2024-03-25T18:23:18.994 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:23:18.994 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:18.995 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:23:18.994130+0000 2024-03-25T18:23:18.995 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:23:18.994130+0000 2024-03-25T18:23:18.995 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:18.995 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:18.995 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:23:18.995 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:23:18.995 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:23:18.995 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:18.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:18.994+0000 7fe479e51640 1 Processor -- start 2024-03-25T18:23:18.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:18.994+0000 7fe479e51640 1 -- start start 2024-03-25T18:23:18.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:18.994+0000 7fe479e51640 10 monclient: init 2024-03-25T18:23:18.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:18.994+0000 7fe479e51640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:18.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:18.994+0000 7fe479e51640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:18.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:18.995+0000 7fe479e51640 10 monclient(hunting): picked mon.noname-a con 0x7fe4740b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T18:23:18.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:18.995+0000 7fe479e51640 10 monclient(hunting): picked mon.noname-c con 0x7fe4740b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:23:18.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:18.995+0000 7fe479e51640 10 monclient(hunting): picked mon.noname-b con 0x7fe47414be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:23:18.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:18.995+0000 7fe479e51640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4740af940 con 0x7fe4740b4d60 2024-03-25T18:23:18.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:18.995+0000 7fe479e51640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4740b0a70 con 0x7fe4740b1730 2024-03-25T18:23:18.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:18.995+0000 7fe479e51640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4740b0d50 con 0x7fe47414be90 2024-03-25T18:23:18.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:18.995+0000 7fe479e51640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:18.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:18.995+0000 7fe479e51640 10 monclient(hunting): authenticate will time out at 8093.181558s 2024-03-25T18:23:19.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.002+0000 7fe473fff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fe47414be90 0x7fe47414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:37232/0 (socket says 172.21.15.63:37232) 2024-03-25T18:23:19.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.002+0000 7fe473fff640 1 -- 172.21.15.63:0/1653824791 learned_addr learned my addr 172.21.15.63:0/1653824791 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:23:19.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.016+0000 7fe4727fc640 1 -- 172.21.15.63:0/1653824791 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543765022 0 0) 0x7fe4740b0d50 con 0x7fe47414be90 2024-03-25T18:23:19.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.016+0000 7fe4727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:19.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.016+0000 7fe4727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:19.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.016+0000 7fe4727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:19.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.016+0000 7fe4727fc640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:23:19.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.016+0000 7fe4727fc640 1 -- 172.21.15.63:0/1653824791 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe45c003520 con 0x7fe47414be90 2024-03-25T18:23:19.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.016+0000 7fe4727fc640 1 -- 172.21.15.63:0/1653824791 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612174960 0 0) 0x7fe4740af940 con 0x7fe4740b4d60 2024-03-25T18:23:19.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.016+0000 7fe4727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:19.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.016+0000 7fe4727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:19.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.016+0000 7fe4727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:19.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.016+0000 7fe4727fc640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:23:19.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.016+0000 7fe4727fc640 1 -- 172.21.15.63:0/1653824791 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4740b0d50 con 0x7fe4740b4d60 2024-03-25T18:23:19.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.021+0000 7fe4727fc640 1 -- 172.21.15.63:0/1653824791 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367688914 0 0) 0x7fe4740b0a70 con 0x7fe4740b1730 2024-03-25T18:23:19.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.021+0000 7fe4727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:19.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.021+0000 7fe4727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:19.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.021+0000 7fe4727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:19.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.021+0000 7fe4727fc640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:23:19.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.021+0000 7fe4727fc640 1 -- 172.21.15.63:0/1653824791 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4740af940 con 0x7fe4740b1730 2024-03-25T18:23:19.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe4727fc640 1 -- 172.21.15.63:0/1653824791 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1524887562 0 0) 0x7fe4740af940 con 0x7fe4740b1730 2024-03-25T18:23:19.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe4727fc640 1 -- 172.21.15.63:0/1653824791 >> v1:172.21.15.63:6790/0 conn(0x7fe47414be90 legacy=0x7fe47414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:19.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe4727fc640 1 -- 172.21.15.63:0/1653824791 >> v1:172.21.15.63:6789/0 conn(0x7fe4740b4d60 legacy=0x7fe47413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:19.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe4727fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:19.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe4727fc640 1 monclient: found mon.noname-c 2024-03-25T18:23:19.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe4727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:19.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe4727fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:23:19.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe4727fc640 1 -- 172.21.15.63:0/1653824791 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4740b1030 con 0x7fe4740b1730 2024-03-25T18:23:19.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe4727fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:19.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe4727fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:19.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe4727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:19.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe4727fc640 1 -- 172.21.15.63:0/1653824791 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe4680015b0 con 0x7fe4740b1730 2024-03-25T18:23:19.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe4727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:19.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe4727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:19.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe4727fc640 10 monclient: dump: 2024-03-25T18:23:19.030 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:19.030 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:19.030 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:19.030 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:19.031 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:19.031 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:19.031 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:19.031 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:19.031 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:19.031 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:19.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe479e51640 5 monclient: authenticate success, global_id 4717 2024-03-25T18:23:19.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.028+0000 7fe479e51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:19.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.043+0000 7fe4727fc640 1 -- 172.21.15.63:0/1653824791 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe468001710 con 0x7fe4740b1730 2024-03-25T18:23:19.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.043+0000 7fe4727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:19.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.043+0000 7fe4727fc640 1 -- 172.21.15.63:0/1653824791 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe468001b60 con 0x7fe4740b1730 2024-03-25T18:23:19.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.043+0000 7fe4727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:19.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.043+0000 7fe4727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:19.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.043+0000 7fe4727fc640 10 monclient: dump: 2024-03-25T18:23:19.044 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:19.044 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:19.044 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:19.044 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:19.044 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:19.044 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:19.045 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:19.045 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:19.045 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:19.045 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:19.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.043+0000 7fe479e51640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:19.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.043+0000 7fe479e51640 4 set_mon_vals no callback set 2024-03-25T18:23:19.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.043+0000 7fe479e51640 10 monclient: shutdown 2024-03-25T18:23:19.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.043+0000 7fe479e51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:19.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.043+0000 7fe479e51640 1 -- 172.21.15.63:0/1653824791 >> v1:172.21.15.81:6789/0 conn(0x7fe4740b1730 legacy=0x7fe4740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:19.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.043+0000 7fe479e51640 1 -- 172.21.15.63:0/1653824791 shutdown_connections 2024-03-25T18:23:19.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.043+0000 7fe479e51640 1 -- 172.21.15.63:0/1653824791 >> 172.21.15.63:0/1653824791 conn(0x7fe4740617a0 msgr2=0x7fe474061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:19.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.044+0000 7fe479e51640 1 -- 172.21.15.63:0/1653824791 shutdown_connections 2024-03-25T18:23:19.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.044+0000 7fe479e51640 1 -- 172.21.15.63:0/1653824791 wait complete. 2024-03-25T18:23:19.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.044+0000 7fe479e51640 10 monclient: build_initial_monmap 2024-03-25T18:23:19.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.044+0000 7fe479e51640 10 monclient: monmap: 2024-03-25T18:23:19.047 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:23:19.047 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:19.047 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:23:19.047 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:23:19.047 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:19.047 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:23:19.047 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:23:19.047 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:23:19.048 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:23:19.048 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:19.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.045+0000 7fe479e51640 1 librados: starting msgr at 2024-03-25T18:23:19.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.045+0000 7fe479e51640 1 librados: starting objecter 2024-03-25T18:23:19.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.045+0000 7fe479e51640 1 Processor -- start 2024-03-25T18:23:19.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.045+0000 7fe479e51640 1 -- start start 2024-03-25T18:23:19.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.045+0000 7fe479e51640 1 librados: setting wanted keys 2024-03-25T18:23:19.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.045+0000 7fe479e51640 1 librados: calling monclient init 2024-03-25T18:23:19.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.045+0000 7fe479e51640 10 monclient: init 2024-03-25T18:23:19.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.046+0000 7fe479e51640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:19.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.046+0000 7fe479e51640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:19.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.046+0000 7fe479e51640 10 monclient(hunting): picked mon.noname-a con 0x7fe4740b1730 addr v1:172.21.15.63:6789/0 2024-03-25T18:23:19.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.046+0000 7fe479e51640 10 monclient(hunting): picked mon.noname-b con 0x7fe4740b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T18:23:19.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.046+0000 7fe479e51640 10 monclient(hunting): picked mon.noname-c con 0x7fe47414be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:23:19.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.046+0000 7fe479e51640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe474141a30 con 0x7fe4740b1730 2024-03-25T18:23:19.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.046+0000 7fe479e51640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4741dafb0 con 0x7fe4740b4d60 2024-03-25T18:23:19.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.046+0000 7fe479e51640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe474141ca0 con 0x7fe47414be90 2024-03-25T18:23:19.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.046+0000 7fe479e51640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:19.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.046+0000 7fe479e51640 10 monclient(hunting): authenticate will time out at 8093.233261s 2024-03-25T18:23:19.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.058+0000 7fe4737fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fe4740b1730 0x7fe4741db490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:41976/0 (socket says 172.21.15.63:41976) 2024-03-25T18:23:19.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.058+0000 7fe4737fe640 1 -- 172.21.15.63:0/3730513616 learned_addr learned my addr 172.21.15.63:0/3730513616 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:23:19.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.064+0000 7fe45bfff640 10 client.?.objecter ms_handle_connect 0x7fe47414be90 2024-03-25T18:23:19.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.064+0000 7fe45bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:19.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.064+0000 7fe45bfff640 10 client.?.objecter ms_handle_connect 0x7fe4740b1730 2024-03-25T18:23:19.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.064+0000 7fe45bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:19.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.070+0000 7fe45bfff640 10 client.?.objecter ms_handle_connect 0x7fe4740b4d60 2024-03-25T18:23:19.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.070+0000 7fe45bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:19.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.072+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810818732 0 0) 0x7fe474141ca0 con 0x7fe47414be90 2024-03-25T18:23:19.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.072+0000 7fe45bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:19.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.072+0000 7fe45bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:19.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.072+0000 7fe45bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:19.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.072+0000 7fe45bfff640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:23:19.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.072+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe460004eb0 con 0x7fe47414be90 2024-03-25T18:23:19.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.079+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034552379 0 0) 0x7fe474141a30 con 0x7fe4740b1730 2024-03-25T18:23:19.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.079+0000 7fe45bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:19.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.079+0000 7fe45bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:19.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.079+0000 7fe45bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:19.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.079+0000 7fe45bfff640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:23:19.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.079+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe474141ca0 con 0x7fe4740b1730 2024-03-25T18:23:19.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.085+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2738850858 0 0) 0x7fe460004eb0 con 0x7fe47414be90 2024-03-25T18:23:19.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.085+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe474141a30 con 0x7fe47414be90 2024-03-25T18:23:19.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.085+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe46c001b30 con 0x7fe47414be90 2024-03-25T18:23:19.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.085+0000 7fe45bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:19.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.085+0000 7fe45bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:19.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.085+0000 7fe45bfff640 10 monclient(hunting): dump: 2024-03-25T18:23:19.086 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:19.087 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:19.087 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:19.087 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:19.087 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:19.087 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:19.087 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:19.087 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:19.087 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:19.087 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:19.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.085+0000 7fe45bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:19.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.089+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870866289 0 0) 0x7fe4741dafb0 con 0x7fe4740b4d60 2024-03-25T18:23:19.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.089+0000 7fe45bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:19.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.089+0000 7fe45bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:19.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.089+0000 7fe45bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:19.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.089+0000 7fe45bfff640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:23:19.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.089+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe460004eb0 con 0x7fe4740b4d60 2024-03-25T18:23:19.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.096+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1388906759 0 0) 0x7fe474141a30 con 0x7fe47414be90 2024-03-25T18:23:19.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.096+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 >> v1:172.21.15.81:6789/0 conn(0x7fe4740b4d60 legacy=0x7fe474142820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:19.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.096+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 >> v1:172.21.15.63:6789/0 conn(0x7fe4740b1730 legacy=0x7fe4741db490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:19.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.096+0000 7fe45bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:23:19.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.096+0000 7fe45bfff640 1 monclient: found mon.c 2024-03-25T18:23:19.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.096+0000 7fe45bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:19.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.096+0000 7fe45bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:23:19.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.096+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47413bb90 con 0x7fe47414be90 2024-03-25T18:23:19.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.096+0000 7fe45bfff640 10 monclient: _finish_auth 0 2024-03-25T18:23:19.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.096+0000 7fe45bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:23:19.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.096+0000 7fe45bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:19.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.096+0000 7fe479e51640 5 monclient: authenticate success, global_id 4694 2024-03-25T18:23:19.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.096+0000 7fe479e51640 10 monclient: _renew_subs 2024-03-25T18:23:19.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.096+0000 7fe479e51640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:23:19.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.096+0000 7fe479e51640 1 -- 172.21.15.63:0/3730513616 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4741dbba0 con 0x7fe47414be90 2024-03-25T18:23:19.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.097+0000 7fe479e51640 10 client.4694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:19.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.097+0000 7fe479e51640 10 monclient: _renew_subs 2024-03-25T18:23:19.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.097+0000 7fe479e51640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:23:19.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.097+0000 7fe479e51640 1 -- 172.21.15.63:0/3730513616 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe47413b920 con 0x7fe47414be90 2024-03-25T18:23:19.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.097+0000 7fe479e51640 1 librados: init done 2024-03-25T18:23:19.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.098+0000 7fe479e51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:19.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.098+0000 7fe479e51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:19.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.098+0000 7fe479e51640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:23:19.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.098+0000 7fe479e51640 1 -- 172.21.15.63:0/3730513616 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4740627b0 con 0x7fe47414be90 2024-03-25T18:23:19.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.417+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe46c002020 con 0x7fe47414be90 2024-03-25T18:23:19.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.417+0000 7fe45bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:19.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.417+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fe46c004d00 con 0x7fe47414be90 2024-03-25T18:23:19.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.417+0000 7fe45bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:19.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.417+0000 7fe45bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:19.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.417+0000 7fe45bfff640 10 monclient: dump: 2024-03-25T18:23:19.419 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:19.419 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:19.419 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:19.419 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:19.419 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:19.419 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:19.419 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:19.420 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:19.420 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:19.420 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.417+0000 7fe471ffb640 4 set_mon_vals no callback set 2024-03-25T18:23:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.418+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fe46c01acd0 con 0x7fe47414be90 2024-03-25T18:23:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.419+0000 7fe4737fe640 10 monclient: get_auth_request con 0x7fe46006de00 auth_method 0 2024-03-25T18:23:19.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.419+0000 7fe45bfff640 10 client.4694.objecter ms_handle_connect 0x7fe46006de00 2024-03-25T18:23:19.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.424+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2076230003 0 0) 0x7fe46c054910 con 0x7fe47414be90 2024-03-25T18:23:19.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.424+0000 7fe45bfff640 10 client.4694.objecter ms_dispatch 0x7fe474000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:23:19.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.424+0000 7fe45bfff640 3 client.4694.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:23:19.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.424+0000 7fe45bfff640 3 client.4694.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:23:19.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.424+0000 7fe45bfff640 20 client.4694.objecter dump_active .. 0 homeless 2024-03-25T18:23:19.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.424+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe46c050250 con 0x7fe47414be90 2024-03-25T18:23:19.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.424+0000 7fe45bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:19.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.424+0000 7fe45bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:19.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.689+0000 7fe479e51640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:19.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.689+0000 7fe479e51640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:19.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.689+0000 7fe479e51640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:23:19.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.689+0000 7fe479e51640 1 -- 172.21.15.63:0/3730513616 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe474069f40 con 0x7fe47414be90 2024-03-25T18:23:19.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.712+0000 7fe45bfff640 1 -- 172.21.15.63:0/3730513616 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 515194931) 0x7fe46c052350 con 0x7fe47414be90 2024-03-25T18:23:19.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.712+0000 7fe45bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:19.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.712+0000 7fe45bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:19.714 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:23:19.714 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":11,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:22:39.912010+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:22:39.912010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:19.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.715+0000 7fe479e51640 10 librados: watch_flush enter 2024-03-25T18:23:19.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.715+0000 7fe479e51640 10 librados: watch_flush exit 2024-03-25T18:23:19.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.715+0000 7fe479e51640 20 client.4694.objecter shutdown clearing up homeless session... 2024-03-25T18:23:19.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.715+0000 7fe479e51640 10 client.4694.objecter successfully canceled tick 2024-03-25T18:23:19.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.715+0000 7fe479e51640 1 -- 172.21.15.63:0/3730513616 >> v1:172.21.15.81:6800/3640340873 conn(0x7fe46006de00 legacy=0x7fe4600702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:19.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.715+0000 7fe479e51640 10 monclient: shutdown 2024-03-25T18:23:19.717 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.715+0000 7fe479e51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:19.717 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.715+0000 7fe479e51640 1 -- 172.21.15.63:0/3730513616 >> v1:172.21.15.63:6790/0 conn(0x7fe47414be90 legacy=0x7fe4741e3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:19.717 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.715+0000 7fe479e51640 1 -- 172.21.15.63:0/3730513616 shutdown_connections 2024-03-25T18:23:19.717 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.715+0000 7fe479e51640 1 -- 172.21.15.63:0/3730513616 >> 172.21.15.63:0/3730513616 conn(0x7fe4740617a0 msgr2=0x7fe47413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:19.717 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.715+0000 7fe479e51640 1 -- 172.21.15.63:0/3730513616 shutdown_connections 2024-03-25T18:23:19.717 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.715+0000 7fe479e51640 1 -- 172.21.15.63:0/3730513616 wait complete. 2024-03-25T18:23:19.717 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:19.716+0000 7fe479e51640 1 librados: shutdown 2024-03-25T18:23:19.730 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:25.155 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:06.847 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:25.165 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:06.857 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:25.202 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:06.894 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:25.218 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:06.910 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:25.222 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:11:06.882 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:25.230 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:06.922 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:25.233 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:11:06.892 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:25.264 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:06.956 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:25.266 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:11:06.927 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:26.537 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:26.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.654+0000 7f07fa63e640 10 monclient: get_monmap_and_config 2024-03-25T18:23:26.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.654+0000 7f07fa63e640 10 monclient: build_initial_monmap 2024-03-25T18:23:26.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.654+0000 7f07fa63e640 10 monclient: monmap: 2024-03-25T18:23:26.656 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:23:26.656 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:26.656 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:23:26.655687+0000 2024-03-25T18:23:26.656 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:23:26.655687+0000 2024-03-25T18:23:26.656 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:26.656 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:26.656 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:23:26.656 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:23:26.656 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:23:26.656 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:26.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.655+0000 7f07fa63e640 1 Processor -- start 2024-03-25T18:23:26.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.655+0000 7f07fa63e640 1 -- start start 2024-03-25T18:23:26.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.655+0000 7f07fa63e640 10 monclient: init 2024-03-25T18:23:26.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.656+0000 7f07fa63e640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:26.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.656+0000 7f07fa63e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:26.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.656+0000 7f07fa63e640 10 monclient(hunting): picked mon.noname-c con 0x7f07f40b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T18:23:26.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.656+0000 7f07fa63e640 10 monclient(hunting): picked mon.noname-a con 0x7f07f40b1730 addr v1:172.21.15.63:6789/0 2024-03-25T18:23:26.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.656+0000 7f07fa63e640 10 monclient(hunting): picked mon.noname-b con 0x7f07f414be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:23:26.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.656+0000 7f07fa63e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f40af940 con 0x7f07f40b1730 2024-03-25T18:23:26.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.656+0000 7f07fa63e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f40b0a70 con 0x7f07f40b4d60 2024-03-25T18:23:26.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.656+0000 7f07fa63e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f40b0d50 con 0x7f07f414be90 2024-03-25T18:23:26.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.656+0000 7f07fa63e640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:26.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.656+0000 7f07fa63e640 10 monclient(hunting): authenticate will time out at 8100.842386s 2024-03-25T18:23:26.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.674+0000 7f07f3fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f07f40b4d60 0x7f07f413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:56048/0 (socket says 172.21.15.63:56048) 2024-03-25T18:23:26.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.674+0000 7f07f3fff640 1 -- 172.21.15.63:0/1673974279 learned_addr learned my addr 172.21.15.63:0/1673974279 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:23:26.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.689+0000 7f07f2ffd640 1 -- 172.21.15.63:0/1673974279 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836878782 0 0) 0x7f07f40b0d50 con 0x7f07f414be90 2024-03-25T18:23:26.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.689+0000 7f07f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:26.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.689+0000 7f07f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:26.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.689+0000 7f07f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:26.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.689+0000 7f07f2ffd640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:23:26.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.689+0000 7f07f2ffd640 1 -- 172.21.15.63:0/1673974279 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07dc003520 con 0x7f07f414be90 2024-03-25T18:23:26.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.690+0000 7f07f2ffd640 1 -- 172.21.15.63:0/1673974279 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248047721 0 0) 0x7f07f40b0a70 con 0x7f07f40b4d60 2024-03-25T18:23:26.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.690+0000 7f07f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:26.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.690+0000 7f07f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:26.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.690+0000 7f07f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:26.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.690+0000 7f07f2ffd640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:23:26.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.690+0000 7f07f2ffd640 1 -- 172.21.15.63:0/1673974279 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07f40b0d50 con 0x7f07f40b4d60 2024-03-25T18:23:26.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.701+0000 7f07f2ffd640 1 -- 172.21.15.63:0/1673974279 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 451588142 0 0) 0x7f07f40b0d50 con 0x7f07f40b4d60 2024-03-25T18:23:26.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.701+0000 7f07f2ffd640 1 -- 172.21.15.63:0/1673974279 >> v1:172.21.15.63:6790/0 conn(0x7f07f414be90 legacy=0x7f07f414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:26.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.701+0000 7f07f2ffd640 1 -- 172.21.15.63:0/1673974279 >> v1:172.21.15.63:6789/0 conn(0x7f07f40b1730 legacy=0x7f07f40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:26.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.701+0000 7f07f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:26.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.701+0000 7f07f2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:23:26.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.701+0000 7f07f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:26.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.702+0000 7f07f2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:23:26.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.702+0000 7f07f2ffd640 1 -- 172.21.15.63:0/1673974279 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f40b1030 con 0x7f07f40b4d60 2024-03-25T18:23:26.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.702+0000 7f07f2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:23:26.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.702+0000 7f07f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:26.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.702+0000 7f07f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:26.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.702+0000 7f07f2ffd640 1 -- 172.21.15.63:0/1673974279 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f07e4001670 con 0x7f07f40b4d60 2024-03-25T18:23:26.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.702+0000 7f07f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:26.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.702+0000 7f07f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:26.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.702+0000 7f07f2ffd640 10 monclient: dump: 2024-03-25T18:23:26.704 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:26.704 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:26.704 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:26.704 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:26.704 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:26.704 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:26.704 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:26.704 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:26.705 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:26.705 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:26.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.702+0000 7f07fa63e640 5 monclient: authenticate success, global_id 4723 2024-03-25T18:23:26.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.702+0000 7f07fa63e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:26.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.708+0000 7f07f2ffd640 1 -- 172.21.15.63:0/1673974279 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07e40017d0 con 0x7f07f40b4d60 2024-03-25T18:23:26.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.708+0000 7f07f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:26.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.708+0000 7f07f2ffd640 1 -- 172.21.15.63:0/1673974279 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f07e4001ba0 con 0x7f07f40b4d60 2024-03-25T18:23:26.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.708+0000 7f07fa63e640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:26.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.708+0000 7f07fa63e640 4 set_mon_vals no callback set 2024-03-25T18:23:26.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.708+0000 7f07f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:26.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.708+0000 7f07f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:26.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.708+0000 7f07f2ffd640 10 monclient: dump: 2024-03-25T18:23:26.710 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:26.710 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:26.710 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:26.710 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:26.710 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:26.710 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:26.710 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:26.711 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:26.711 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:26.711 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:26.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.709+0000 7f07fa63e640 10 monclient: shutdown 2024-03-25T18:23:26.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.709+0000 7f07fa63e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:26.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.709+0000 7f07fa63e640 1 -- 172.21.15.63:0/1673974279 >> v1:172.21.15.81:6789/0 conn(0x7f07f40b4d60 legacy=0x7f07f413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:26.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.709+0000 7f07fa63e640 1 -- 172.21.15.63:0/1673974279 shutdown_connections 2024-03-25T18:23:26.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.709+0000 7f07fa63e640 1 -- 172.21.15.63:0/1673974279 >> 172.21.15.63:0/1673974279 conn(0x7f07f40617a0 msgr2=0x7f07f4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:26.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.709+0000 7f07fa63e640 1 -- 172.21.15.63:0/1673974279 shutdown_connections 2024-03-25T18:23:26.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.709+0000 7f07fa63e640 1 -- 172.21.15.63:0/1673974279 wait complete. 2024-03-25T18:23:26.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.710+0000 7f07fa63e640 10 monclient: build_initial_monmap 2024-03-25T18:23:26.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.710+0000 7f07fa63e640 10 monclient: monmap: 2024-03-25T18:23:26.713 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:23:26.713 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:26.713 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:23:26.713 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:23:26.713 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:26.713 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:23:26.713 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:23:26.713 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:23:26.713 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:23:26.714 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:26.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.711+0000 7f07fa63e640 1 librados: starting msgr at 2024-03-25T18:23:26.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.711+0000 7f07fa63e640 1 librados: starting objecter 2024-03-25T18:23:26.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.711+0000 7f07fa63e640 1 Processor -- start 2024-03-25T18:23:26.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.711+0000 7f07fa63e640 1 -- start start 2024-03-25T18:23:26.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.711+0000 7f07fa63e640 1 librados: setting wanted keys 2024-03-25T18:23:26.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.711+0000 7f07fa63e640 1 librados: calling monclient init 2024-03-25T18:23:26.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.711+0000 7f07fa63e640 10 monclient: init 2024-03-25T18:23:26.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.712+0000 7f07fa63e640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:26.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.712+0000 7f07fa63e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:26.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.712+0000 7f07fa63e640 10 monclient(hunting): picked mon.noname-b con 0x7f07f40b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:23:26.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.712+0000 7f07fa63e640 10 monclient(hunting): picked mon.noname-a con 0x7f07f40b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T18:23:26.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.712+0000 7f07fa63e640 10 monclient(hunting): picked mon.noname-c con 0x7f07f414be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:23:26.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.712+0000 7f07fa63e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f41d79d0 con 0x7f07f40b4d60 2024-03-25T18:23:26.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.712+0000 7f07fa63e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f41d8600 con 0x7f07f40b1730 2024-03-25T18:23:26.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.712+0000 7f07fa63e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f4141f90 con 0x7f07f414be90 2024-03-25T18:23:26.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.712+0000 7f07fa63e640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:26.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.712+0000 7f07fa63e640 10 monclient(hunting): authenticate will time out at 8100.898899s 2024-03-25T18:23:26.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.715+0000 7f07f8bdb640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f07f414be90 0x7f07f412fef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:41834/0 (socket says 172.21.15.63:41834) 2024-03-25T18:23:26.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.715+0000 7f07f8bdb640 1 -- 172.21.15.63:0/4120554688 learned_addr learned my addr 172.21.15.63:0/4120554688 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:23:26.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.718+0000 7f07f0ff9640 10 client.?.objecter ms_handle_connect 0x7f07f414be90 2024-03-25T18:23:26.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.718+0000 7f07f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:26.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.733+0000 7f07f0ff9640 10 client.?.objecter ms_handle_connect 0x7f07f40b1730 2024-03-25T18:23:26.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.733+0000 7f07f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:26.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.735+0000 7f07f0ff9640 10 client.?.objecter ms_handle_connect 0x7f07f40b4d60 2024-03-25T18:23:26.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.735+0000 7f07f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:26.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.737+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642758323 0 0) 0x7f07f41d8600 con 0x7f07f40b1730 2024-03-25T18:23:26.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.737+0000 7f07f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:26.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.737+0000 7f07f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:26.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.737+0000 7f07f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:26.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.737+0000 7f07f0ff9640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:23:26.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.737+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07e0004eb0 con 0x7f07f40b1730 2024-03-25T18:23:26.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.741+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720733018 0 0) 0x7f07f41d79d0 con 0x7f07f40b4d60 2024-03-25T18:23:26.743 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.741+0000 7f07f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:26.743 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.741+0000 7f07f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:26.743 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.741+0000 7f07f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:26.743 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.741+0000 7f07f0ff9640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:23:26.743 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.741+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07f41d8600 con 0x7f07f40b4d60 2024-03-25T18:23:26.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.749+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733979744 0 0) 0x7f07f4141f90 con 0x7f07f414be90 2024-03-25T18:23:26.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.749+0000 7f07f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:26.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.749+0000 7f07f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:26.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.749+0000 7f07f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:26.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.749+0000 7f07f0ff9640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:23:26.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.749+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07f41d79d0 con 0x7f07f414be90 2024-03-25T18:23:26.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.750+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3225437555 0 0) 0x7f07e0004eb0 con 0x7f07f40b1730 2024-03-25T18:23:26.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.750+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07f4141f90 con 0x7f07f40b1730 2024-03-25T18:23:26.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.750+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f07e4001ae0 con 0x7f07f40b1730 2024-03-25T18:23:26.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.750+0000 7f07f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:26.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.750+0000 7f07f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:26.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.750+0000 7f07f0ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:26.756 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:26.757 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:26.757 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:26.757 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:26.757 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:26.757 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:26.757 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:26.757 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:26.757 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:26.757 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:26.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.750+0000 7f07f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:26.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.759+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1005731919 0 0) 0x7f07f41d8600 con 0x7f07f40b4d60 2024-03-25T18:23:26.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.759+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07e0004eb0 con 0x7f07f40b4d60 2024-03-25T18:23:26.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.759+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f07e80018b0 con 0x7f07f40b4d60 2024-03-25T18:23:26.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.759+0000 7f07f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:26.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.759+0000 7f07f0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:26.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.759+0000 7f07f0ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:26.760 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:26.760 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:26.760 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:26.760 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:26.760 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:26.761 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:26.761 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:26.761 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:26.761 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:26.761 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:26.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 504399421 0 0) 0x7f07f4141f90 con 0x7f07f40b1730 2024-03-25T18:23:26.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 >> v1:172.21.15.63:6790/0 conn(0x7f07f414be90 legacy=0x7f07f412fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:26.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 >> v1:172.21.15.63:6789/0 conn(0x7f07f40b4d60 legacy=0x7f07f4141880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:26.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07f0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:26.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07f0ff9640 1 monclient: found mon.b 2024-03-25T18:23:26.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:26.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07f0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:23:26.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f4142120 con 0x7f07f40b1730 2024-03-25T18:23:26.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07f0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:26.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07f0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:26.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:26.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07fa63e640 5 monclient: authenticate success, global_id 4726 2024-03-25T18:23:26.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07fa63e640 10 monclient: _renew_subs 2024-03-25T18:23:26.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07fa63e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:23:26.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07fa63e640 1 -- 172.21.15.63:0/4120554688 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f4130600 con 0x7f07f40b1730 2024-03-25T18:23:26.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07fa63e640 10 client.4726.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:26.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07fa63e640 10 monclient: _renew_subs 2024-03-25T18:23:26.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07fa63e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:23:26.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07fa63e640 1 -- 172.21.15.63:0/4120554688 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f4143ae0 con 0x7f07f40b1730 2024-03-25T18:23:26.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.764+0000 7f07fa63e640 1 librados: init done 2024-03-25T18:23:26.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.765+0000 7f07fa63e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:26.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.765+0000 7f07fa63e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:26.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.765+0000 7f07fa63e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:23:26.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:26.765+0000 7f07fa63e640 1 -- 172.21.15.63:0/4120554688 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f41d7830 con 0x7f07f40b1730 2024-03-25T18:23:27.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.031+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07e4004a90 con 0x7f07f40b1730 2024-03-25T18:23:27.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.031+0000 7f07f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:27.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.031+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f07e4004c00 con 0x7f07f40b1730 2024-03-25T18:23:27.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.031+0000 7f07f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:27.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.031+0000 7f07f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:27.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.031+0000 7f07f0ff9640 10 monclient: dump: 2024-03-25T18:23:27.033 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:27.033 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:27.033 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:27.033 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:27.033 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:27.033 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:27.033 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:27.033 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:27.033 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:27.033 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:27.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.031+0000 7f07f27fc640 4 set_mon_vals no callback set 2024-03-25T18:23:27.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.032+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f07e401abd0 con 0x7f07f40b1730 2024-03-25T18:23:27.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.033+0000 7f07f37fe640 10 monclient: get_auth_request con 0x7f07e006df70 auth_method 0 2024-03-25T18:23:27.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.033+0000 7f07f0ff9640 10 client.4726.objecter ms_handle_connect 0x7f07e006df70 2024-03-25T18:23:27.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.040+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2076230003 0 0) 0x7f07e4054720 con 0x7f07f40b1730 2024-03-25T18:23:27.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.040+0000 7f07f0ff9640 10 client.4726.objecter ms_dispatch 0x7f07f4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:23:27.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.040+0000 7f07f0ff9640 3 client.4726.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:23:27.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.040+0000 7f07f0ff9640 3 client.4726.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:23:27.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.040+0000 7f07f0ff9640 20 client.4726.objecter dump_active .. 0 homeless 2024-03-25T18:23:27.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.040+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f07e4050230 con 0x7f07f40b1730 2024-03-25T18:23:27.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.041+0000 7f07f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:27.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.041+0000 7f07f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:27.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.287+0000 7f07fa63e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:27.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.288+0000 7f07fa63e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:27.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.288+0000 7f07fa63e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:23:27.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.288+0000 7f07fa63e640 1 -- 172.21.15.63:0/4120554688 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07f40627b0 con 0x7f07f40b1730 2024-03-25T18:23:27.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.430+0000 7f07f0ff9640 1 -- 172.21.15.63:0/4120554688 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 515194931) 0x7f07e4052160 con 0x7f07f40b1730 2024-03-25T18:23:27.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.430+0000 7f07f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:27.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.430+0000 7f07f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:27.432 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:23:27.432 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":11,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:22:39.912010+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:22:39.912010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:27.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.432+0000 7f07fa63e640 10 librados: watch_flush enter 2024-03-25T18:23:27.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.432+0000 7f07fa63e640 10 librados: watch_flush exit 2024-03-25T18:23:27.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.432+0000 7f07fa63e640 20 client.4726.objecter shutdown clearing up homeless session... 2024-03-25T18:23:27.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.432+0000 7f07fa63e640 10 client.4726.objecter successfully canceled tick 2024-03-25T18:23:27.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.432+0000 7f07fa63e640 1 -- 172.21.15.63:0/4120554688 >> v1:172.21.15.81:6800/3640340873 conn(0x7f07e006df70 legacy=0x7f07e0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.432+0000 7f07fa63e640 10 monclient: shutdown 2024-03-25T18:23:27.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.432+0000 7f07fa63e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:27.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.432+0000 7f07fa63e640 1 -- 172.21.15.63:0/4120554688 >> v1:172.21.15.81:6789/0 conn(0x7f07f40b1730 legacy=0x7f07f41d7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.432+0000 7f07fa63e640 1 -- 172.21.15.63:0/4120554688 shutdown_connections 2024-03-25T18:23:27.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.432+0000 7f07fa63e640 1 -- 172.21.15.63:0/4120554688 >> 172.21.15.63:0/4120554688 conn(0x7f07f40617a0 msgr2=0x7f07f40b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:27.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.432+0000 7f07fa63e640 1 -- 172.21.15.63:0/4120554688 shutdown_connections 2024-03-25T18:23:27.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.432+0000 7f07fa63e640 1 -- 172.21.15.63:0/4120554688 wait complete. 2024-03-25T18:23:27.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:27.433+0000 7f07fa63e640 1 librados: shutdown 2024-03-25T18:23:27.446 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:34.252 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:34.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.367+0000 7f86bb4f6640 10 monclient: get_monmap_and_config 2024-03-25T18:23:34.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.367+0000 7f86bb4f6640 10 monclient: build_initial_monmap 2024-03-25T18:23:34.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.367+0000 7f86bb4f6640 10 monclient: monmap: 2024-03-25T18:23:34.368 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:23:34.368 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:34.369 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:23:34.368213+0000 2024-03-25T18:23:34.369 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:23:34.368213+0000 2024-03-25T18:23:34.369 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:34.369 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:34.369 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:23:34.369 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:23:34.369 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:23:34.369 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:34.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.367+0000 7f86bb4f6640 1 Processor -- start 2024-03-25T18:23:34.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.368+0000 7f86bb4f6640 1 -- start start 2024-03-25T18:23:34.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.368+0000 7f86bb4f6640 10 monclient: init 2024-03-25T18:23:34.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.368+0000 7f86bb4f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:34.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.368+0000 7f86bb4f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:34.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.368+0000 7f86bb4f6640 10 monclient(hunting): picked mon.noname-c con 0x7f86b40a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:23:34.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.368+0000 7f86bb4f6640 10 monclient(hunting): picked mon.noname-a con 0x7f86b413f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:23:34.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.368+0000 7f86bb4f6640 10 monclient(hunting): picked mon.noname-b con 0x7f86b41430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:23:34.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.368+0000 7f86bb4f6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b4142520 con 0x7f86b413f710 2024-03-25T18:23:34.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.368+0000 7f86bb4f6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b40b28a0 con 0x7f86b40a98f0 2024-03-25T18:23:34.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.368+0000 7f86bb4f6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b40b2a30 con 0x7f86b41430c0 2024-03-25T18:23:34.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.369+0000 7f86bb4f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:34.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.369+0000 7f86bb4f6640 10 monclient(hunting): authenticate will time out at 8108.555302s 2024-03-25T18:23:34.380 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.378+0000 7f86b9292640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f86b40a98f0 0x7f86b40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:35458/0 (socket says 172.21.15.63:35458) 2024-03-25T18:23:34.380 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.378+0000 7f86b9292640 1 -- 172.21.15.63:0/3858427853 learned_addr learned my addr 172.21.15.63:0/3858427853 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:23:34.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.389+0000 7f86abfff640 1 -- 172.21.15.63:0/3858427853 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678071542 0 0) 0x7f86b40b28a0 con 0x7f86b40a98f0 2024-03-25T18:23:34.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.389+0000 7f86abfff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:34.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.389+0000 7f86abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:34.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.389+0000 7f86abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:34.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.389+0000 7f86abfff640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:23:34.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.389+0000 7f86abfff640 1 -- 172.21.15.63:0/3858427853 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a0003520 con 0x7f86b40a98f0 2024-03-25T18:23:34.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86abfff640 1 -- 172.21.15.63:0/3858427853 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 197983845 0 0) 0x7f86a0003520 con 0x7f86b40a98f0 2024-03-25T18:23:34.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86abfff640 1 -- 172.21.15.63:0/3858427853 >> v1:172.21.15.63:6790/0 conn(0x7f86b41430c0 legacy=0x7f86b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:34.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86abfff640 1 -- 172.21.15.63:0/3858427853 >> v1:172.21.15.63:6789/0 conn(0x7f86b413f710 legacy=0x7f86b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:34.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86abfff640 10 monclient: _finish_hunting 0 2024-03-25T18:23:34.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86abfff640 1 monclient: found mon.noname-c 2024-03-25T18:23:34.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:34.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86abfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:23:34.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86abfff640 1 -- 172.21.15.63:0/3858427853 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b40b44c0 con 0x7f86b40a98f0 2024-03-25T18:23:34.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86abfff640 10 monclient: _finish_auth 0 2024-03-25T18:23:34.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86abfff640 10 monclient: _check_auth_tickets 2024-03-25T18:23:34.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:34.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86abfff640 1 -- 172.21.15.63:0/3858427853 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f86b0001690 con 0x7f86b40a98f0 2024-03-25T18:23:34.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86bb4f6640 5 monclient: authenticate success, global_id 4729 2024-03-25T18:23:34.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86bb4f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:34.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:34.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86abfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:34.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86abfff640 10 monclient: dump: 2024-03-25T18:23:34.403 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:34.403 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:34.403 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:34.403 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:34.403 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:34.403 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:34.403 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:34.403 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:34.403 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:34.403 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:34.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.399+0000 7f86bb4f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:34.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.404+0000 7f86abfff640 1 -- 172.21.15.63:0/3858427853 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86b00017f0 con 0x7f86b40a98f0 2024-03-25T18:23:34.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.404+0000 7f86abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:34.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.404+0000 7f86abfff640 1 -- 172.21.15.63:0/3858427853 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f86b0001c00 con 0x7f86b40a98f0 2024-03-25T18:23:34.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.404+0000 7f86bb4f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:34.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.404+0000 7f86bb4f6640 4 set_mon_vals no callback set 2024-03-25T18:23:34.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.404+0000 7f86abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:34.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.404+0000 7f86abfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:34.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.404+0000 7f86abfff640 10 monclient: dump: 2024-03-25T18:23:34.406 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:34.406 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:34.406 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:34.406 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:34.406 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:34.406 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:34.406 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:34.406 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:34.406 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:34.406 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:34.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.405+0000 7f86bb4f6640 10 monclient: shutdown 2024-03-25T18:23:34.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.405+0000 7f86bb4f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:34.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.405+0000 7f86bb4f6640 1 -- 172.21.15.63:0/3858427853 >> v1:172.21.15.81:6789/0 conn(0x7f86b40a98f0 legacy=0x7f86b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:34.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.405+0000 7f86bb4f6640 1 -- 172.21.15.63:0/3858427853 shutdown_connections 2024-03-25T18:23:34.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.405+0000 7f86bb4f6640 1 -- 172.21.15.63:0/3858427853 >> 172.21.15.63:0/3858427853 conn(0x7f86b40b7ea0 msgr2=0x7f86b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:34.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.405+0000 7f86bb4f6640 1 -- 172.21.15.63:0/3858427853 shutdown_connections 2024-03-25T18:23:34.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.405+0000 7f86bb4f6640 1 -- 172.21.15.63:0/3858427853 wait complete. 2024-03-25T18:23:34.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.406+0000 7f86bb4f6640 10 monclient: build_initial_monmap 2024-03-25T18:23:34.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.406+0000 7f86bb4f6640 10 monclient: monmap: 2024-03-25T18:23:34.408 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:23:34.408 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:34.408 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:23:34.408 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:23:34.408 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:34.409 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:23:34.409 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:23:34.409 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:23:34.409 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:23:34.409 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:34.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.406+0000 7f86bb4f6640 1 librados: starting msgr at 2024-03-25T18:23:34.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.406+0000 7f86bb4f6640 1 librados: starting objecter 2024-03-25T18:23:34.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.406+0000 7f86bb4f6640 1 Processor -- start 2024-03-25T18:23:34.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.406+0000 7f86bb4f6640 1 -- start start 2024-03-25T18:23:34.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.406+0000 7f86bb4f6640 1 librados: setting wanted keys 2024-03-25T18:23:34.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.406+0000 7f86bb4f6640 1 librados: calling monclient init 2024-03-25T18:23:34.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.406+0000 7f86bb4f6640 10 monclient: init 2024-03-25T18:23:34.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.407+0000 7f86bb4f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:34.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.407+0000 7f86bb4f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:34.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.407+0000 7f86bb4f6640 10 monclient(hunting): picked mon.noname-a con 0x7f86b40a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:23:34.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.407+0000 7f86bb4f6640 10 monclient(hunting): picked mon.noname-b con 0x7f86b413f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:23:34.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.407+0000 7f86bb4f6640 10 monclient(hunting): picked mon.noname-c con 0x7f86b41430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:23:34.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.407+0000 7f86bb4f6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b40ab9e0 con 0x7f86b40a98f0 2024-03-25T18:23:34.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.407+0000 7f86bb4f6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b4138c20 con 0x7f86b413f710 2024-03-25T18:23:34.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.407+0000 7f86bb4f6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b4138f00 con 0x7f86b41430c0 2024-03-25T18:23:34.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.407+0000 7f86bb4f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:34.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.407+0000 7f86bb4f6640 10 monclient(hunting): authenticate will time out at 8108.594178s 2024-03-25T18:23:34.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.428+0000 7f86b9a93640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f86b41430c0 0x7f86b41e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:57400/0 (socket says 172.21.15.63:57400) 2024-03-25T18:23:34.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.428+0000 7f86b9a93640 1 -- 172.21.15.63:0/1547684511 learned_addr learned my addr 172.21.15.63:0/1547684511 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:23:34.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.428+0000 7f86a9ffb640 10 client.?.objecter ms_handle_connect 0x7f86b41430c0 2024-03-25T18:23:34.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.428+0000 7f86a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:34.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.434+0000 7f86a9ffb640 10 client.?.objecter ms_handle_connect 0x7f86b40a98f0 2024-03-25T18:23:34.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.434+0000 7f86a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:34.445 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.444+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794702639 0 0) 0x7f86b40ab9e0 con 0x7f86b40a98f0 2024-03-25T18:23:34.445 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.444+0000 7f86a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:34.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.444+0000 7f86a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:34.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.444+0000 7f86a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:34.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.444+0000 7f86a9ffb640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:23:34.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.444+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8698004eb0 con 0x7f86b40a98f0 2024-03-25T18:23:34.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.446+0000 7f86a9ffb640 10 client.?.objecter ms_handle_connect 0x7f86b413f710 2024-03-25T18:23:34.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.446+0000 7f86a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:34.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.447+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418118193 0 0) 0x7f86b4138f00 con 0x7f86b41430c0 2024-03-25T18:23:34.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.447+0000 7f86a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:34.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.447+0000 7f86a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:34.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.447+0000 7f86a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:34.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.447+0000 7f86a9ffb640 10 monclient(hunting): my global_id is 4706 2024-03-25T18:23:34.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.447+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b40ab9e0 con 0x7f86b41430c0 2024-03-25T18:23:34.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.457+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2786092061 0 0) 0x7f8698004eb0 con 0x7f86b40a98f0 2024-03-25T18:23:34.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.457+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86b4138f00 con 0x7f86b40a98f0 2024-03-25T18:23:34.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.457+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f86b0001bc0 con 0x7f86b40a98f0 2024-03-25T18:23:34.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.457+0000 7f86a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:34.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.457+0000 7f86a9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:34.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.457+0000 7f86a9ffb640 10 monclient(hunting): dump: 2024-03-25T18:23:34.459 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:34.459 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:34.459 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:34.459 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:34.459 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:34.459 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:34.459 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:34.459 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:34.459 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:34.460 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:34.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.457+0000 7f86a9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:34.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.463+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699753428 0 0) 0x7f86b4138c20 con 0x7f86b413f710 2024-03-25T18:23:34.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.463+0000 7f86a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:34.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.463+0000 7f86a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:34.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.463+0000 7f86a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:34.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.463+0000 7f86a9ffb640 10 monclient(hunting): my global_id is 4732 2024-03-25T18:23:34.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.463+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8698004eb0 con 0x7f86b413f710 2024-03-25T18:23:34.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2201639876 0 0) 0x7f86b4138f00 con 0x7f86b40a98f0 2024-03-25T18:23:34.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 >> v1:172.21.15.63:6790/0 conn(0x7f86b41430c0 legacy=0x7f86b41e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:34.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 >> v1:172.21.15.81:6789/0 conn(0x7f86b413f710 legacy=0x7f86b41d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:34.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86a9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:23:34.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86a9ffb640 1 monclient: found mon.a 2024-03-25T18:23:34.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:34.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86a9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:34.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b414e7a0 con 0x7f86b40a98f0 2024-03-25T18:23:34.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86a9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:23:34.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86a9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:23:34.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:34.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86bb4f6640 5 monclient: authenticate success, global_id 4692 2024-03-25T18:23:34.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86bb4f6640 10 monclient: _renew_subs 2024-03-25T18:23:34.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86bb4f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:34.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86bb4f6640 1 -- 172.21.15.63:0/1547684511 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b41386d0 con 0x7f86b40a98f0 2024-03-25T18:23:34.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86bb4f6640 10 client.4692.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:34.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86bb4f6640 10 monclient: _renew_subs 2024-03-25T18:23:34.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86bb4f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:34.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86bb4f6640 1 -- 172.21.15.63:0/1547684511 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b41d8060 con 0x7f86b40a98f0 2024-03-25T18:23:34.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.476+0000 7f86bb4f6640 1 librados: init done 2024-03-25T18:23:34.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.477+0000 7f86bb4f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:34.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.477+0000 7f86bb4f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:34.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.477+0000 7f86bb4f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:34.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.477+0000 7f86bb4f6640 1 -- 172.21.15.63:0/1547684511 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b40627b0 con 0x7f86b40a98f0 2024-03-25T18:23:34.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.795+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86b0004a90 con 0x7f86b40a98f0 2024-03-25T18:23:34.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.795+0000 7f86a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:34.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.795+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f86b0004da0 con 0x7f86b40a98f0 2024-03-25T18:23:34.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.795+0000 7f86a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:34.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.795+0000 7f86a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:34.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.795+0000 7f86a9ffb640 10 monclient: dump: 2024-03-25T18:23:34.796 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:34.796 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:34.796 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:34.797 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:34.797 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:34.797 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:34.797 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:34.797 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:34.797 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:34.797 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:34.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.795+0000 7f86ab7fe640 4 set_mon_vals no callback set 2024-03-25T18:23:34.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.796+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f86b001ad70 con 0x7f86b40a98f0 2024-03-25T18:23:34.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.796+0000 7f86b8a91640 10 monclient: get_auth_request con 0x7f869806de00 auth_method 0 2024-03-25T18:23:34.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.797+0000 7f86a9ffb640 10 client.4692.objecter ms_handle_connect 0x7f869806de00 2024-03-25T18:23:34.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.802+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2076230003 0 0) 0x7f86b00548b0 con 0x7f86b40a98f0 2024-03-25T18:23:34.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.802+0000 7f86a9ffb640 10 client.4692.objecter ms_dispatch 0x7f86b4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:23:34.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.802+0000 7f86a9ffb640 3 client.4692.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:23:34.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.802+0000 7f86a9ffb640 3 client.4692.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:23:34.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.802+0000 7f86a9ffb640 20 client.4692.objecter dump_active .. 0 homeless 2024-03-25T18:23:34.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.802+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86b00501f0 con 0x7f86b40a98f0 2024-03-25T18:23:34.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.802+0000 7f86a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:34.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:34.802+0000 7f86a9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:35.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.013+0000 7f86bb4f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:35.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.013+0000 7f86bb4f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:35.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.013+0000 7f86bb4f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:35.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.013+0000 7f86bb4f6640 1 -- 172.21.15.63:0/1547684511 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86b4069f40 con 0x7f86b40a98f0 2024-03-25T18:23:35.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.057+0000 7f86a9ffb640 1 -- 172.21.15.63:0/1547684511 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 515194931) 0x7f86b00522f0 con 0x7f86b40a98f0 2024-03-25T18:23:35.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.057+0000 7f86a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:35.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.057+0000 7f86a9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:35.059 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:23:35.059 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":11,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:22:39.912010+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:22:39.912010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:35.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.060+0000 7f86bb4f6640 10 librados: watch_flush enter 2024-03-25T18:23:35.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.060+0000 7f86bb4f6640 10 librados: watch_flush exit 2024-03-25T18:23:35.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.060+0000 7f86bb4f6640 20 client.4692.objecter shutdown clearing up homeless session... 2024-03-25T18:23:35.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.060+0000 7f86bb4f6640 10 client.4692.objecter successfully canceled tick 2024-03-25T18:23:35.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.060+0000 7f86bb4f6640 1 -- 172.21.15.63:0/1547684511 >> v1:172.21.15.81:6800/3640340873 conn(0x7f869806de00 legacy=0x7f86980702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:35.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.060+0000 7f86bb4f6640 10 monclient: shutdown 2024-03-25T18:23:35.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.060+0000 7f86bb4f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:35.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.060+0000 7f86bb4f6640 1 -- 172.21.15.63:0/1547684511 >> v1:172.21.15.63:6789/0 conn(0x7f86b40a98f0 legacy=0x7f86b4137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:35.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.060+0000 7f86bb4f6640 1 -- 172.21.15.63:0/1547684511 shutdown_connections 2024-03-25T18:23:35.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.060+0000 7f86bb4f6640 1 -- 172.21.15.63:0/1547684511 >> 172.21.15.63:0/1547684511 conn(0x7f86b40b7ea0 msgr2=0x7f86b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:35.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.060+0000 7f86bb4f6640 1 -- 172.21.15.63:0/1547684511 shutdown_connections 2024-03-25T18:23:35.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.060+0000 7f86bb4f6640 1 -- 172.21.15.63:0/1547684511 wait complete. 2024-03-25T18:23:35.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:35.060+0000 7f86bb4f6640 1 librados: shutdown 2024-03-25T18:23:35.073 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:41.879 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:41.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:41.990+0000 7fdaf3b3a640 10 monclient: get_monmap_and_config 2024-03-25T18:23:41.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:41.990+0000 7fdaf3b3a640 10 monclient: build_initial_monmap 2024-03-25T18:23:41.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:41.990+0000 7fdaf3b3a640 10 monclient: monmap: 2024-03-25T18:23:41.992 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:23:41.992 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:41.992 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:23:41.992032+0000 2024-03-25T18:23:41.992 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:23:41.992032+0000 2024-03-25T18:23:41.992 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:41.992 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:41.992 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:23:41.993 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:23:41.993 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:23:41.993 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:41.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:41.991+0000 7fdaf3b3a640 1 Processor -- start 2024-03-25T18:23:41.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:41.991+0000 7fdaf3b3a640 1 -- start start 2024-03-25T18:23:41.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:41.991+0000 7fdaf3b3a640 10 monclient: init 2024-03-25T18:23:41.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:41.992+0000 7fdaf3b3a640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:41.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:41.992+0000 7fdaf3b3a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:41.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:41.992+0000 7fdaf3b3a640 10 monclient(hunting): picked mon.noname-c con 0x7fdaec0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T18:23:41.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:41.992+0000 7fdaf3b3a640 10 monclient(hunting): picked mon.noname-a con 0x7fdaec13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:23:41.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:41.992+0000 7fdaf3b3a640 10 monclient(hunting): picked mon.noname-b con 0x7fdaec1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:23:41.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:41.992+0000 7fdaf3b3a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec142520 con 0x7fdaec13f710 2024-03-25T18:23:41.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:41.992+0000 7fdaf3b3a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec0b28a0 con 0x7fdaec0a98f0 2024-03-25T18:23:41.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:41.992+0000 7fdaf3b3a640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec0b2a30 con 0x7fdaec1430c0 2024-03-25T18:23:41.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:41.992+0000 7fdaf3b3a640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:41.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:41.992+0000 7fdaf3b3a640 10 monclient(hunting): authenticate will time out at 8116.178972s 2024-03-25T18:23:42.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.005+0000 7fdaf18d6640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fdaec0a98f0 0x7fdaec0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:52808/0 (socket says 172.21.15.63:52808) 2024-03-25T18:23:42.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.005+0000 7fdaf18d6640 1 -- 172.21.15.63:0/3999492264 learned_addr learned my addr 172.21.15.63:0/3999492264 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:23:42.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.016+0000 7fdaf08d4640 1 -- 172.21.15.63:0/3999492264 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892495520 0 0) 0x7fdaec0b28a0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.016+0000 7fdaf08d4640 10 monclient(hunting): _negotiate 2024-03-25T18:23:42.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.016+0000 7fdaf08d4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:42.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.016+0000 7fdaf08d4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:42.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.016+0000 7fdaf08d4640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:23:42.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.016+0000 7fdaf08d4640 1 -- 172.21.15.63:0/3999492264 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad8003520 con 0x7fdaec0a98f0 2024-03-25T18:23:42.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf08d4640 1 -- 172.21.15.63:0/3999492264 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2306694399 0 0) 0x7fdad8003520 con 0x7fdaec0a98f0 2024-03-25T18:23:42.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf08d4640 1 -- 172.21.15.63:0/3999492264 >> v1:172.21.15.63:6790/0 conn(0x7fdaec1430c0 legacy=0x7fdaec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:42.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf08d4640 1 -- 172.21.15.63:0/3999492264 >> v1:172.21.15.63:6789/0 conn(0x7fdaec13f710 legacy=0x7fdaec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:42.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf08d4640 10 monclient: _finish_hunting 0 2024-03-25T18:23:42.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf08d4640 1 monclient: found mon.noname-c 2024-03-25T18:23:42.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf08d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:42.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf08d4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:23:42.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf08d4640 1 -- 172.21.15.63:0/3999492264 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec0b44c0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf08d4640 10 monclient: _finish_auth 0 2024-03-25T18:23:42.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf08d4640 10 monclient: _check_auth_tickets 2024-03-25T18:23:42.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf08d4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:42.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf08d4640 1 -- 172.21.15.63:0/3999492264 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fdae8001690 con 0x7fdaec0a98f0 2024-03-25T18:23:42.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf08d4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:42.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf08d4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:42.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf08d4640 10 monclient: dump: 2024-03-25T18:23:42.025 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:42.025 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:42.025 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:42.025 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:42.025 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:42.025 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:42.025 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:42.025 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:42.026 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:42.026 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:42.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf3b3a640 5 monclient: authenticate success, global_id 4735 2024-03-25T18:23:42.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.022+0000 7fdaf3b3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:42.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.029+0000 7fdaf08d4640 1 -- 172.21.15.63:0/3999492264 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdae80017f0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.029+0000 7fdaf08d4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:42.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.029+0000 7fdaf08d4640 1 -- 172.21.15.63:0/3999492264 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fdae8001c00 con 0x7fdaec0a98f0 2024-03-25T18:23:42.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.029+0000 7fdaf08d4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:42.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.029+0000 7fdaf08d4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:42.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.029+0000 7fdaf08d4640 10 monclient: dump: 2024-03-25T18:23:42.031 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:42.031 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:42.031 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:42.031 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:42.031 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:42.031 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:42.031 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:42.031 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:42.032 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:42.032 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:42.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.029+0000 7fdaf3b3a640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:42.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.029+0000 7fdaf3b3a640 4 set_mon_vals no callback set 2024-03-25T18:23:42.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.030+0000 7fdaf3b3a640 10 monclient: shutdown 2024-03-25T18:23:42.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.030+0000 7fdaf3b3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:42.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.030+0000 7fdaf3b3a640 1 -- 172.21.15.63:0/3999492264 >> v1:172.21.15.81:6789/0 conn(0x7fdaec0a98f0 legacy=0x7fdaec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:42.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.030+0000 7fdaf3b3a640 1 -- 172.21.15.63:0/3999492264 shutdown_connections 2024-03-25T18:23:42.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.030+0000 7fdaf3b3a640 1 -- 172.21.15.63:0/3999492264 >> 172.21.15.63:0/3999492264 conn(0x7fdaec0b7ea0 msgr2=0x7fdaec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:42.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.031+0000 7fdaf3b3a640 1 -- 172.21.15.63:0/3999492264 shutdown_connections 2024-03-25T18:23:42.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.031+0000 7fdaf3b3a640 1 -- 172.21.15.63:0/3999492264 wait complete. 2024-03-25T18:23:42.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.032+0000 7fdaf3b3a640 10 monclient: build_initial_monmap 2024-03-25T18:23:42.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.032+0000 7fdaf3b3a640 10 monclient: monmap: 2024-03-25T18:23:42.033 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:23:42.033 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:42.033 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:23:42.034 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:23:42.034 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:42.034 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:23:42.034 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:23:42.034 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:23:42.034 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:23:42.034 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:42.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.032+0000 7fdaf3b3a640 1 librados: starting msgr at 2024-03-25T18:23:42.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.032+0000 7fdaf3b3a640 1 librados: starting objecter 2024-03-25T18:23:42.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.032+0000 7fdaf3b3a640 1 Processor -- start 2024-03-25T18:23:42.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.032+0000 7fdaf3b3a640 1 -- start start 2024-03-25T18:23:42.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.032+0000 7fdaf3b3a640 1 librados: setting wanted keys 2024-03-25T18:23:42.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.032+0000 7fdaf3b3a640 1 librados: calling monclient init 2024-03-25T18:23:42.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.032+0000 7fdaf3b3a640 10 monclient: init 2024-03-25T18:23:42.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.033+0000 7fdaf3b3a640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:42.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.033+0000 7fdaf3b3a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:42.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.033+0000 7fdaf3b3a640 10 monclient(hunting): picked mon.noname-c con 0x7fdaec0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:23:42.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.033+0000 7fdaf3b3a640 10 monclient(hunting): picked mon.noname-b con 0x7fdaec13f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:23:42.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.033+0000 7fdaf3b3a640 10 monclient(hunting): picked mon.noname-a con 0x7fdaec1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T18:23:42.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.033+0000 7fdaf3b3a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec1db7d0 con 0x7fdaec1430c0 2024-03-25T18:23:42.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.033+0000 7fdaf3b3a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec0b1420 con 0x7fdaec13f710 2024-03-25T18:23:42.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.033+0000 7fdaf3b3a640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec0b15b0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.033+0000 7fdaf3b3a640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:42.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.033+0000 7fdaf3b3a640 10 monclient(hunting): authenticate will time out at 8116.219775s 2024-03-25T18:23:42.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.053+0000 7fdaf18d6640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fdaec0a98f0 0x7fdaec1daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:49758/0 (socket says 172.21.15.63:49758) 2024-03-25T18:23:42.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.053+0000 7fdaf18d6640 1 -- 172.21.15.63:0/1846776595 learned_addr learned my addr 172.21.15.63:0/1846776595 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:23:42.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.058+0000 7fdae27fc640 10 client.?.objecter ms_handle_connect 0x7fdaec0a98f0 2024-03-25T18:23:42.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.058+0000 7fdae27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:42.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.063+0000 7fdae27fc640 10 client.?.objecter ms_handle_connect 0x7fdaec1430c0 2024-03-25T18:23:42.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.063+0000 7fdae27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:42.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.065+0000 7fdae27fc640 10 client.?.objecter ms_handle_connect 0x7fdaec13f710 2024-03-25T18:23:42.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.065+0000 7fdae27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:42.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.074+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649353887 0 0) 0x7fdaec0b1420 con 0x7fdaec13f710 2024-03-25T18:23:42.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.075+0000 7fdae27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:42.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.075+0000 7fdae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:42.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.075+0000 7fdae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:42.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.075+0000 7fdae27fc640 10 monclient(hunting): my global_id is 4738 2024-03-25T18:23:42.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.075+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad0004eb0 con 0x7fdaec13f710 2024-03-25T18:23:42.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.075+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317889523 0 0) 0x7fdaec1db7d0 con 0x7fdaec1430c0 2024-03-25T18:23:42.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.075+0000 7fdae27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:42.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.075+0000 7fdae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:42.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.075+0000 7fdae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:42.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.075+0000 7fdae27fc640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:23:42.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.075+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaec0b1420 con 0x7fdaec1430c0 2024-03-25T18:23:42.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.076+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287944239 0 0) 0x7fdaec0b15b0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.076+0000 7fdae27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:42.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.076+0000 7fdae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:42.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.076+0000 7fdae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:42.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.076+0000 7fdae27fc640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:23:42.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.076+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaec1db7d0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.101+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 877306116 0 0) 0x7fdaec0b1420 con 0x7fdaec1430c0 2024-03-25T18:23:42.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.101+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdaec0b15b0 con 0x7fdaec1430c0 2024-03-25T18:23:42.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.101+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fdae4001b10 con 0x7fdaec1430c0 2024-03-25T18:23:42.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.101+0000 7fdae27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:42.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.101+0000 7fdae27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:42.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.101+0000 7fdae27fc640 10 monclient(hunting): dump: 2024-03-25T18:23:42.103 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:42.103 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:42.103 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:42.103 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:42.103 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:42.103 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:42.103 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:42.103 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:42.103 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:42.103 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:42.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.101+0000 7fdae27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:42.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.113+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2535658477 0 0) 0x7fdad0004eb0 con 0x7fdaec13f710 2024-03-25T18:23:42.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.113+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdaec0b1420 con 0x7fdaec13f710 2024-03-25T18:23:42.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.113+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fdadc0018c0 con 0x7fdaec13f710 2024-03-25T18:23:42.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.113+0000 7fdae27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:42.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.113+0000 7fdae27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:42.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.113+0000 7fdae27fc640 10 monclient(hunting): dump: 2024-03-25T18:23:42.215 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:42.215 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:42.215 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:42.215 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:42.215 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:42.215 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:42.216 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:42.216 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:42.216 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:42.216 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:42.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.113+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4082166808 0 0) 0x7fdaec1db7d0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.113+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdad0004eb0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.113+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fdae8001bc0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.113+0000 7fdae27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:42.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.113+0000 7fdae27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:42.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.113+0000 7fdae27fc640 10 monclient(hunting): dump: 2024-03-25T18:23:42.216 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:42.216 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:42.216 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:42.216 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:42.217 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:42.217 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:42.217 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:42.217 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:42.217 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:42.217 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:42.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.121+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3481772816 0 0) 0x7fdad0004eb0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 >> v1:172.21.15.81:6789/0 conn(0x7fdaec13f710 legacy=0x7fdaec0b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:42.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 >> v1:172.21.15.63:6789/0 conn(0x7fdaec1430c0 legacy=0x7fdaec1d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:42.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdae27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:42.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdae27fc640 1 monclient: found mon.c 2024-03-25T18:23:42.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdae27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:42.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdae27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:23:42.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec0b1740 con 0x7fdaec0a98f0 2024-03-25T18:23:42.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdae27fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:42.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdae27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:42.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdae27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:42.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdaf3b3a640 5 monclient: authenticate success, global_id 4712 2024-03-25T18:23:42.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdaf3b3a640 10 monclient: _renew_subs 2024-03-25T18:23:42.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdaf3b3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:23:42.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdaf3b3a640 1 -- 172.21.15.63:0/1846776595 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaec1db1a0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdaf3b3a640 10 client.4712.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:42.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdaf3b3a640 10 monclient: _renew_subs 2024-03-25T18:23:42.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdaf3b3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:23:42.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdaf3b3a640 1 -- 172.21.15.63:0/1846776595 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaec0b08f0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.122+0000 7fdaf3b3a640 1 librados: init done 2024-03-25T18:23:42.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.123+0000 7fdaf3b3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:42.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.123+0000 7fdaf3b3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:42.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.123+0000 7fdaf3b3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:23:42.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.123+0000 7fdaf3b3a640 1 -- 172.21.15.63:0/1846776595 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaec0627b0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.452+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdae8004a90 con 0x7fdaec0a98f0 2024-03-25T18:23:42.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.452+0000 7fdae27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:42.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.452+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7fdae8004da0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.452+0000 7fdae27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:42.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.452+0000 7fdae27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:42.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.452+0000 7fdae27fc640 10 monclient: dump: 2024-03-25T18:23:42.454 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:42.454 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:42.454 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:42.454 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:42.454 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:42.454 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:42.454 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:42.454 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:42.454 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:42.454 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:42.455 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.452+0000 7fdae3fff640 4 set_mon_vals no callback set 2024-03-25T18:23:42.455 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.453+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7fdae801ad70 con 0x7fdaec0a98f0 2024-03-25T18:23:42.455 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.454+0000 7fdaf10d5640 10 monclient: get_auth_request con 0x7fdad006e160 auth_method 0 2024-03-25T18:23:42.455 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.454+0000 7fdae27fc640 10 client.4712.objecter ms_handle_connect 0x7fdad006e160 2024-03-25T18:23:42.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.458+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2076230003 0 0) 0x7fdae8054890 con 0x7fdaec0a98f0 2024-03-25T18:23:42.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.458+0000 7fdae27fc640 10 client.4712.objecter ms_dispatch 0x7fdaec000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:23:42.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.458+0000 7fdae27fc640 3 client.4712.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:23:42.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.458+0000 7fdae27fc640 3 client.4712.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:23:42.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.458+0000 7fdae27fc640 20 client.4712.objecter dump_active .. 0 homeless 2024-03-25T18:23:42.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.458+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdae80503a0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.458+0000 7fdae27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:42.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.458+0000 7fdae27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:42.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.684+0000 7fdaf3b3a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:42.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.685+0000 7fdaf3b3a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:42.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.685+0000 7fdaf3b3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T18:23:42.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.685+0000 7fdaf3b3a640 1 -- 172.21.15.63:0/1846776595 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdaec0abae0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.724+0000 7fdae27fc640 1 -- 172.21.15.63:0/1846776595 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 515194931) 0x7fdae80522d0 con 0x7fdaec0a98f0 2024-03-25T18:23:42.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.725+0000 7fdae27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:42.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.725+0000 7fdae27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:42.726 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:23:42.726 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":11,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:22:39.912010+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:22:39.912010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:42.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.727+0000 7fdaf3b3a640 10 librados: watch_flush enter 2024-03-25T18:23:42.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.727+0000 7fdaf3b3a640 10 librados: watch_flush exit 2024-03-25T18:23:42.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.727+0000 7fdaf3b3a640 20 client.4712.objecter shutdown clearing up homeless session... 2024-03-25T18:23:42.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.727+0000 7fdaf3b3a640 10 client.4712.objecter successfully canceled tick 2024-03-25T18:23:42.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.727+0000 7fdaf3b3a640 1 -- 172.21.15.63:0/1846776595 >> v1:172.21.15.81:6800/3640340873 conn(0x7fdad006e160 legacy=0x7fdad0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:42.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.727+0000 7fdaf3b3a640 10 monclient: shutdown 2024-03-25T18:23:42.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.727+0000 7fdaf3b3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:42.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.727+0000 7fdaf3b3a640 1 -- 172.21.15.63:0/1846776595 >> v1:172.21.15.63:6790/0 conn(0x7fdaec0a98f0 legacy=0x7fdaec1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:42.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.727+0000 7fdaf3b3a640 1 -- 172.21.15.63:0/1846776595 shutdown_connections 2024-03-25T18:23:42.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.727+0000 7fdaf3b3a640 1 -- 172.21.15.63:0/1846776595 >> 172.21.15.63:0/1846776595 conn(0x7fdaec0b7ea0 msgr2=0x7fdaec13c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:42.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.727+0000 7fdaf3b3a640 1 -- 172.21.15.63:0/1846776595 shutdown_connections 2024-03-25T18:23:42.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.727+0000 7fdaf3b3a640 1 -- 172.21.15.63:0/1846776595 wait complete. 2024-03-25T18:23:42.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:42.727+0000 7fdaf3b3a640 1 librados: shutdown 2024-03-25T18:23:42.740 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:43.871 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:25.563 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:43.873 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:25.565 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:43.894 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:11:25.554 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:43.900 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:25.592 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:43.900 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:11:25.560 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:43.903 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:25.595 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:43.906 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:25.598 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:43.919 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:11:25.579 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:43.928 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:25.620 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:45.208 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:26.900 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:45.218 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:26.910 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:45.264 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:11:26.925 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:45.265 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:26.958 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:45.268 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:26.960 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:45.270 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:11:26.930 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:45.271 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:26.963 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:45.290 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:11:26.950 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:45.301 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:26.994 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:49.546 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:49.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.657+0000 7f9b424b3640 10 monclient: get_monmap_and_config 2024-03-25T18:23:49.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.657+0000 7f9b424b3640 10 monclient: build_initial_monmap 2024-03-25T18:23:49.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.657+0000 7f9b424b3640 10 monclient: monmap: 2024-03-25T18:23:49.659 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:23:49.659 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:49.659 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:23:49.658835+0000 2024-03-25T18:23:49.659 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:23:49.658835+0000 2024-03-25T18:23:49.659 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:49.659 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:49.659 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:23:49.659 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:23:49.659 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:23:49.660 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:49.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.658+0000 7f9b424b3640 1 Processor -- start 2024-03-25T18:23:49.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.658+0000 7f9b424b3640 1 -- start start 2024-03-25T18:23:49.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.658+0000 7f9b424b3640 10 monclient: init 2024-03-25T18:23:49.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.659+0000 7f9b424b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:49.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.659+0000 7f9b424b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:49.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.659+0000 7f9b424b3640 10 monclient(hunting): picked mon.noname-a con 0x7f9b3c0a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:23:49.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.659+0000 7f9b424b3640 10 monclient(hunting): picked mon.noname-c con 0x7f9b3c13f710 addr v1:172.21.15.81:6789/0 2024-03-25T18:23:49.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.659+0000 7f9b424b3640 10 monclient(hunting): picked mon.noname-b con 0x7f9b3c1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T18:23:49.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.659+0000 7f9b424b3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c142520 con 0x7f9b3c0a98f0 2024-03-25T18:23:49.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.659+0000 7f9b424b3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c0b28a0 con 0x7f9b3c13f710 2024-03-25T18:23:49.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.659+0000 7f9b424b3640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c0b2a30 con 0x7f9b3c1430c0 2024-03-25T18:23:49.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.659+0000 7f9b424b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:49.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.659+0000 7f9b424b3640 10 monclient(hunting): authenticate will time out at 8123.845971s 2024-03-25T18:23:49.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.664+0000 7f9b40a50640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f9b3c1430c0 0x7f9b3c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:47614/0 (socket says 172.21.15.63:47614) 2024-03-25T18:23:49.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.664+0000 7f9b40a50640 1 -- 172.21.15.63:0/970068007 learned_addr learned my addr 172.21.15.63:0/970068007 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:23:49.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.676+0000 7f9b3affd640 1 -- 172.21.15.63:0/970068007 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787061380 0 0) 0x7f9b3c0b2a30 con 0x7f9b3c1430c0 2024-03-25T18:23:49.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.676+0000 7f9b3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:49.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.676+0000 7f9b3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:49.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.676+0000 7f9b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:49.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.676+0000 7f9b3affd640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:23:49.678 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.676+0000 7f9b3affd640 1 -- 172.21.15.63:0/970068007 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b24003520 con 0x7f9b3c1430c0 2024-03-25T18:23:49.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b3affd640 1 -- 172.21.15.63:0/970068007 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1760774914 0 0) 0x7f9b24003520 con 0x7f9b3c1430c0 2024-03-25T18:23:49.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b3affd640 1 -- 172.21.15.63:0/970068007 >> v1:172.21.15.81:6789/0 conn(0x7f9b3c13f710 legacy=0x7f9b3c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:49.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b3affd640 1 -- 172.21.15.63:0/970068007 >> v1:172.21.15.63:6789/0 conn(0x7f9b3c0a98f0 legacy=0x7f9b3c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:49.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b3affd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:49.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b3affd640 1 monclient: found mon.noname-b 2024-03-25T18:23:49.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:49.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b3affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T18:23:49.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b3affd640 1 -- 172.21.15.63:0/970068007 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3c0b44c0 con 0x7f9b3c1430c0 2024-03-25T18:23:49.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b3affd640 10 monclient: _finish_auth 0 2024-03-25T18:23:49.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b3affd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:49.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:49.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b3affd640 1 -- 172.21.15.63:0/970068007 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9b34001a10 con 0x7f9b3c1430c0 2024-03-25T18:23:49.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:49.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b3affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:49.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b3affd640 10 monclient: dump: 2024-03-25T18:23:49.689 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:49.689 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:49.689 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:49.689 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:49.689 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:49.689 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:49.689 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:49.689 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:49.689 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:49.689 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:49.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b424b3640 5 monclient: authenticate success, global_id 4715 2024-03-25T18:23:49.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.686+0000 7f9b424b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:49.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.700+0000 7f9b3affd640 1 -- 172.21.15.63:0/970068007 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b34001b70 con 0x7f9b3c1430c0 2024-03-25T18:23:49.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.700+0000 7f9b3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:49.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.700+0000 7f9b3affd640 1 -- 172.21.15.63:0/970068007 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9b34001f80 con 0x7f9b3c1430c0 2024-03-25T18:23:49.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.700+0000 7f9b3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:49.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.700+0000 7f9b3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:49.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.700+0000 7f9b3affd640 10 monclient: dump: 2024-03-25T18:23:49.702 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:49.702 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:49.702 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:49.702 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:49.702 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:49.702 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:49.702 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:49.703 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:49.703 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:49.703 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:49.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.700+0000 7f9b424b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:49.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.700+0000 7f9b424b3640 4 set_mon_vals no callback set 2024-03-25T18:23:49.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.701+0000 7f9b424b3640 10 monclient: shutdown 2024-03-25T18:23:49.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.701+0000 7f9b424b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:49.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.701+0000 7f9b424b3640 1 -- 172.21.15.63:0/970068007 >> v1:172.21.15.63:6790/0 conn(0x7f9b3c1430c0 legacy=0x7f9b3c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:49.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.701+0000 7f9b424b3640 1 -- 172.21.15.63:0/970068007 shutdown_connections 2024-03-25T18:23:49.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.701+0000 7f9b424b3640 1 -- 172.21.15.63:0/970068007 >> 172.21.15.63:0/970068007 conn(0x7f9b3c0b7ea0 msgr2=0x7f9b3c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:49.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.701+0000 7f9b424b3640 1 -- 172.21.15.63:0/970068007 shutdown_connections 2024-03-25T18:23:49.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.701+0000 7f9b424b3640 1 -- 172.21.15.63:0/970068007 wait complete. 2024-03-25T18:23:49.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.702+0000 7f9b424b3640 10 monclient: build_initial_monmap 2024-03-25T18:23:49.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.702+0000 7f9b424b3640 10 monclient: monmap: 2024-03-25T18:23:49.706 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:23:49.706 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:49.706 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:23:49.706 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:23:49.706 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:49.706 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:23:49.706 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:23:49.706 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:23:49.706 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:23:49.706 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:49.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.702+0000 7f9b424b3640 1 librados: starting msgr at 2024-03-25T18:23:49.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.702+0000 7f9b424b3640 1 librados: starting objecter 2024-03-25T18:23:49.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.702+0000 7f9b424b3640 1 Processor -- start 2024-03-25T18:23:49.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.702+0000 7f9b424b3640 1 -- start start 2024-03-25T18:23:49.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.703+0000 7f9b424b3640 1 librados: setting wanted keys 2024-03-25T18:23:49.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.703+0000 7f9b424b3640 1 librados: calling monclient init 2024-03-25T18:23:49.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.703+0000 7f9b424b3640 10 monclient: init 2024-03-25T18:23:49.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.704+0000 7f9b424b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:49.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.704+0000 7f9b424b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:49.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.704+0000 7f9b424b3640 10 monclient(hunting): picked mon.noname-c con 0x7f9b3c0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:23:49.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.704+0000 7f9b424b3640 10 monclient(hunting): picked mon.noname-a con 0x7f9b3c13f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:23:49.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.704+0000 7f9b424b3640 10 monclient(hunting): picked mon.noname-b con 0x7f9b3c1430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:23:49.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.704+0000 7f9b424b3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c1369d0 con 0x7f9b3c13f710 2024-03-25T18:23:49.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.704+0000 7f9b424b3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c133a60 con 0x7f9b3c1430c0 2024-03-25T18:23:49.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.704+0000 7f9b424b3640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c0ab9e0 con 0x7f9b3c0a98f0 2024-03-25T18:23:49.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.704+0000 7f9b424b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:49.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.704+0000 7f9b424b3640 10 monclient(hunting): authenticate will time out at 8123.890691s 2024-03-25T18:23:49.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.709+0000 7f9b40a50640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f9b3c1430c0 0x7f9b3c133350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:37724/0 (socket says 172.21.15.63:37724) 2024-03-25T18:23:49.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.709+0000 7f9b40a50640 1 -- 172.21.15.63:0/4002150302 learned_addr learned my addr 172.21.15.63:0/4002150302 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:23:49.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.711+0000 7f9b38ff9640 10 client.?.objecter ms_handle_connect 0x7f9b3c1430c0 2024-03-25T18:23:49.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.711+0000 7f9b38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:49.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.712+0000 7f9b38ff9640 10 client.?.objecter ms_handle_connect 0x7f9b3c0a98f0 2024-03-25T18:23:49.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.712+0000 7f9b38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:49.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.719+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900060204 0 0) 0x7f9b3c133a60 con 0x7f9b3c1430c0 2024-03-25T18:23:49.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.719+0000 7f9b38ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:49.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.719+0000 7f9b38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:49.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.719+0000 7f9b38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:49.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.719+0000 7f9b38ff9640 10 monclient(hunting): my global_id is 4744 2024-03-25T18:23:49.721 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.719+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b28004eb0 con 0x7f9b3c1430c0 2024-03-25T18:23:49.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.722+0000 7f9b38ff9640 10 client.?.objecter ms_handle_connect 0x7f9b3c13f710 2024-03-25T18:23:49.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.722+0000 7f9b38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:49.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.729+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677477255 0 0) 0x7f9b3c0ab9e0 con 0x7f9b3c0a98f0 2024-03-25T18:23:49.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.729+0000 7f9b38ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:49.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.729+0000 7f9b38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:49.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.729+0000 7f9b38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:49.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.729+0000 7f9b38ff9640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:23:49.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.729+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b3c133a60 con 0x7f9b3c0a98f0 2024-03-25T18:23:49.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.732+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3102375897 0 0) 0x7f9b28004eb0 con 0x7f9b3c1430c0 2024-03-25T18:23:49.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.732+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b3c0ab9e0 con 0x7f9b3c1430c0 2024-03-25T18:23:49.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.732+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9b34001f40 con 0x7f9b3c1430c0 2024-03-25T18:23:49.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.732+0000 7f9b38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:49.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.732+0000 7f9b38ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:49.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.732+0000 7f9b38ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:49.734 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:49.734 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:49.734 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:49.734 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:49.734 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:49.734 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:49.735 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:49.735 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:49.735 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:49.735 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:49.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.732+0000 7f9b38ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:49.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.736+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201489055 0 0) 0x7f9b3c1369d0 con 0x7f9b3c13f710 2024-03-25T18:23:49.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.736+0000 7f9b38ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:49.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.736+0000 7f9b38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:49.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.736+0000 7f9b38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:49.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.736+0000 7f9b38ff9640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:23:49.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.736+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b28004eb0 con 0x7f9b3c13f710 2024-03-25T18:23:49.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4127033562 0 0) 0x7f9b3c0ab9e0 con 0x7f9b3c1430c0 2024-03-25T18:23:49.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 >> v1:172.21.15.63:6790/0 conn(0x7f9b3c0a98f0 legacy=0x7f9b3c132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:49.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 >> v1:172.21.15.63:6789/0 conn(0x7f9b3c13f710 legacy=0x7f9b3c1362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:49.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b38ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:49.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b38ff9640 1 monclient: found mon.b 2024-03-25T18:23:49.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b38ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:49.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b38ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:23:49.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3c0abcc0 con 0x7f9b3c1430c0 2024-03-25T18:23:49.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b38ff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:49.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b38ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:49.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:49.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b424b3640 5 monclient: authenticate success, global_id 4744 2024-03-25T18:23:49.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b424b3640 10 monclient: _renew_subs 2024-03-25T18:23:49.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b424b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:23:49.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b424b3640 1 -- 172.21.15.63:0/4002150302 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3c0abfa0 con 0x7f9b3c1430c0 2024-03-25T18:23:49.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b424b3640 10 client.4744.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:49.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b424b3640 10 monclient: _renew_subs 2024-03-25T18:23:49.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b424b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:23:49.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b424b3640 1 -- 172.21.15.63:0/4002150302 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3c137e40 con 0x7f9b3c1430c0 2024-03-25T18:23:49.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.743+0000 7f9b424b3640 1 librados: init done 2024-03-25T18:23:49.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.744+0000 7f9b424b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:49.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.744+0000 7f9b424b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:49.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.744+0000 7f9b424b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:23:49.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:49.744+0000 7f9b424b3640 1 -- 172.21.15.63:0/4002150302 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3c0627b0 con 0x7f9b3c1430c0 2024-03-25T18:23:50.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.022+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b34004e10 con 0x7f9b3c1430c0 2024-03-25T18:23:50.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.022+0000 7f9b38ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:50.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.022+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f9b34005120 con 0x7f9b3c1430c0 2024-03-25T18:23:50.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.022+0000 7f9b38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:50.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.022+0000 7f9b38ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:50.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.022+0000 7f9b38ff9640 10 monclient: dump: 2024-03-25T18:23:50.024 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:50.024 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:50.024 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:50.024 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:50.025 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:50.025 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:50.025 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:50.025 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:50.025 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:50.025 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:50.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.022+0000 7f9b3a7fc640 4 set_mon_vals no callback set 2024-03-25T18:23:50.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.024+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f9b3401b0f0 con 0x7f9b3c1430c0 2024-03-25T18:23:50.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.024+0000 7f9b3bfff640 10 monclient: get_auth_request con 0x7f9b2806ddb0 auth_method 0 2024-03-25T18:23:50.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.024+0000 7f9b38ff9640 10 client.4744.objecter ms_handle_connect 0x7f9b2806ddb0 2024-03-25T18:23:50.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.031+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2076230003 0 0) 0x7f9b34054bf0 con 0x7f9b3c1430c0 2024-03-25T18:23:50.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.031+0000 7f9b38ff9640 10 client.4744.objecter ms_dispatch 0x7f9b3c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:23:50.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.031+0000 7f9b38ff9640 3 client.4744.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:23:50.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.031+0000 7f9b38ff9640 3 client.4744.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:23:50.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.031+0000 7f9b38ff9640 20 client.4744.objecter dump_active .. 0 homeless 2024-03-25T18:23:50.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.032+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b34050700 con 0x7f9b3c1430c0 2024-03-25T18:23:50.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.032+0000 7f9b38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:50.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.032+0000 7f9b38ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:50.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.290+0000 7f9b424b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:50.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.290+0000 7f9b424b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:50.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.290+0000 7f9b424b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:23:50.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.290+0000 7f9b424b3640 1 -- 172.21.15.63:0/4002150302 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b3c139000 con 0x7f9b3c1430c0 2024-03-25T18:23:50.322 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.321+0000 7f9b38ff9640 1 -- 172.21.15.63:0/4002150302 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 515194931) 0x7f9b34052630 con 0x7f9b3c1430c0 2024-03-25T18:23:50.322 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.321+0000 7f9b38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:50.322 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.321+0000 7f9b38ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:50.323 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:23:50.323 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":11,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:22:39.912010+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:22:39.912010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:50.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.323+0000 7f9b424b3640 10 librados: watch_flush enter 2024-03-25T18:23:50.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.323+0000 7f9b424b3640 10 librados: watch_flush exit 2024-03-25T18:23:50.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.323+0000 7f9b424b3640 20 client.4744.objecter shutdown clearing up homeless session... 2024-03-25T18:23:50.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.323+0000 7f9b424b3640 10 client.4744.objecter successfully canceled tick 2024-03-25T18:23:50.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.323+0000 7f9b424b3640 1 -- 172.21.15.63:0/4002150302 >> v1:172.21.15.81:6800/3640340873 conn(0x7f9b2806ddb0 legacy=0x7f9b28070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.323+0000 7f9b424b3640 10 monclient: shutdown 2024-03-25T18:23:50.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.323+0000 7f9b424b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:50.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.323+0000 7f9b424b3640 1 -- 172.21.15.63:0/4002150302 >> v1:172.21.15.81:6789/0 conn(0x7f9b3c1430c0 legacy=0x7f9b3c133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.323+0000 7f9b424b3640 1 -- 172.21.15.63:0/4002150302 shutdown_connections 2024-03-25T18:23:50.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.323+0000 7f9b424b3640 1 -- 172.21.15.63:0/4002150302 >> 172.21.15.63:0/4002150302 conn(0x7f9b3c0b7ea0 msgr2=0x7f9b3c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:50.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.323+0000 7f9b424b3640 1 -- 172.21.15.63:0/4002150302 shutdown_connections 2024-03-25T18:23:50.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.323+0000 7f9b424b3640 1 -- 172.21.15.63:0/4002150302 wait complete. 2024-03-25T18:23:50.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:50.324+0000 7f9b424b3640 1 librados: shutdown 2024-03-25T18:23:50.336 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:57.143 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:57.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.256+0000 7f70d01a8640 10 monclient: get_monmap_and_config 2024-03-25T18:23:57.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.256+0000 7f70d01a8640 10 monclient: build_initial_monmap 2024-03-25T18:23:57.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.256+0000 7f70d01a8640 10 monclient: monmap: 2024-03-25T18:23:57.257 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:23:57.257 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:57.257 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:23:57.257285+0000 2024-03-25T18:23:57.257 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:23:57.257285+0000 2024-03-25T18:23:57.257 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:57.257 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:57.257 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:23:57.258 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:23:57.258 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:23:57.258 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:57.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.256+0000 7f70d01a8640 1 Processor -- start 2024-03-25T18:23:57.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.257+0000 7f70d01a8640 1 -- start start 2024-03-25T18:23:57.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.257+0000 7f70d01a8640 10 monclient: init 2024-03-25T18:23:57.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.257+0000 7f70d01a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:57.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.257+0000 7f70d01a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:57.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.257+0000 7f70d01a8640 10 monclient(hunting): picked mon.noname-a con 0x7f70c80b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T18:23:57.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.257+0000 7f70d01a8640 10 monclient(hunting): picked mon.noname-c con 0x7f70c80b1730 addr v1:172.21.15.81:6789/0 2024-03-25T18:23:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.257+0000 7f70d01a8640 10 monclient(hunting): picked mon.noname-b con 0x7f70c814be90 addr v1:172.21.15.63:6790/0 2024-03-25T18:23:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.257+0000 7f70d01a8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c80af940 con 0x7f70c80b4d60 2024-03-25T18:23:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.257+0000 7f70d01a8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c80b0a70 con 0x7f70c80b1730 2024-03-25T18:23:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.258+0000 7f70d01a8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c80b0d50 con 0x7f70c814be90 2024-03-25T18:23:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.258+0000 7f70d01a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.258+0000 7f70d01a8640 10 monclient(hunting): authenticate will time out at 8131.444303s 2024-03-25T18:23:57.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.262+0000 7f70cd743640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f70c80b1730 0x7f70c80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:42342/0 (socket says 172.21.15.63:42342) 2024-03-25T18:23:57.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.262+0000 7f70cd743640 1 -- 172.21.15.63:0/2741835307 learned_addr learned my addr 172.21.15.63:0/2741835307 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:23:57.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.271+0000 7f70ccf42640 1 -- 172.21.15.63:0/2741835307 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833843649 0 0) 0x7f70c80b0a70 con 0x7f70c80b1730 2024-03-25T18:23:57.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.271+0000 7f70ccf42640 10 monclient(hunting): _negotiate 2024-03-25T18:23:57.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.271+0000 7f70ccf42640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:57.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.271+0000 7f70ccf42640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:57.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.271+0000 7f70ccf42640 10 monclient(hunting): my global_id is 4747 2024-03-25T18:23:57.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.271+0000 7f70ccf42640 1 -- 172.21.15.63:0/2741835307 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b4003520 con 0x7f70c80b1730 2024-03-25T18:23:57.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70ccf42640 1 -- 172.21.15.63:0/2741835307 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3875398811 0 0) 0x7f70b4003520 con 0x7f70c80b1730 2024-03-25T18:23:57.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70ccf42640 1 -- 172.21.15.63:0/2741835307 >> v1:172.21.15.63:6790/0 conn(0x7f70c814be90 legacy=0x7f70c814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:57.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70ccf42640 1 -- 172.21.15.63:0/2741835307 >> v1:172.21.15.63:6789/0 conn(0x7f70c80b4d60 legacy=0x7f70c813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:23:57.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70ccf42640 10 monclient: _finish_hunting 0 2024-03-25T18:23:57.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70ccf42640 1 monclient: found mon.noname-c 2024-03-25T18:23:57.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70ccf42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:57.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70ccf42640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:23:57.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70ccf42640 1 -- 172.21.15.63:0/2741835307 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c80b1030 con 0x7f70c80b1730 2024-03-25T18:23:57.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70ccf42640 10 monclient: _finish_auth 0 2024-03-25T18:23:57.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70ccf42640 10 monclient: _check_auth_tickets 2024-03-25T18:23:57.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70ccf42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:57.280 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70ccf42640 1 -- 172.21.15.63:0/2741835307 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f70b8001690 con 0x7f70c80b1730 2024-03-25T18:23:57.280 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70ccf42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:57.280 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70ccf42640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:57.280 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70ccf42640 10 monclient: dump: 2024-03-25T18:23:57.280 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:57.280 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:57.280 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:57.280 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:57.280 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:57.280 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:57.280 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:57.280 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:57.280 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:57.280 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:57.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70d01a8640 5 monclient: authenticate success, global_id 4747 2024-03-25T18:23:57.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.277+0000 7f70d01a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:57.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.288+0000 7f70ccf42640 1 -- 172.21.15.63:0/2741835307 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70b8001860 con 0x7f70c80b1730 2024-03-25T18:23:57.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.288+0000 7f70ccf42640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:57.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.288+0000 7f70ccf42640 1 -- 172.21.15.63:0/2741835307 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f70b8001cb0 con 0x7f70c80b1730 2024-03-25T18:23:57.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.288+0000 7f70ccf42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:57.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.288+0000 7f70ccf42640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:57.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.288+0000 7f70ccf42640 10 monclient: dump: 2024-03-25T18:23:57.289 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:57.289 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:57.289 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:57.290 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:57.290 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:57.290 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:57.290 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:57.290 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:57.290 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:57.290 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:57.290 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.288+0000 7f70d01a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:57.290 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.288+0000 7f70d01a8640 4 set_mon_vals no callback set 2024-03-25T18:23:57.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.289+0000 7f70d01a8640 10 monclient: shutdown 2024-03-25T18:23:57.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.289+0000 7f70d01a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:57.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.289+0000 7f70d01a8640 1 -- 172.21.15.63:0/2741835307 >> v1:172.21.15.81:6789/0 conn(0x7f70c80b1730 legacy=0x7f70c80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:57.292 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.289+0000 7f70d01a8640 1 -- 172.21.15.63:0/2741835307 shutdown_connections 2024-03-25T18:23:57.292 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.289+0000 7f70d01a8640 1 -- 172.21.15.63:0/2741835307 >> 172.21.15.63:0/2741835307 conn(0x7f70c80617a0 msgr2=0x7f70c8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:57.292 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.289+0000 7f70d01a8640 1 -- 172.21.15.63:0/2741835307 shutdown_connections 2024-03-25T18:23:57.292 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.289+0000 7f70d01a8640 1 -- 172.21.15.63:0/2741835307 wait complete. 2024-03-25T18:23:57.292 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.290+0000 7f70d01a8640 10 monclient: build_initial_monmap 2024-03-25T18:23:57.292 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.290+0000 7f70d01a8640 10 monclient: monmap: 2024-03-25T18:23:57.292 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:23:57.292 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:57.292 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:23:57.292 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:23:57.292 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:57.292 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:23:57.292 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:23:57.292 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:23:57.292 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:23:57.292 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:57.293 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.290+0000 7f70d01a8640 1 librados: starting msgr at 2024-03-25T18:23:57.293 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.290+0000 7f70d01a8640 1 librados: starting objecter 2024-03-25T18:23:57.293 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.290+0000 7f70d01a8640 1 Processor -- start 2024-03-25T18:23:57.293 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.290+0000 7f70d01a8640 1 -- start start 2024-03-25T18:23:57.293 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.290+0000 7f70d01a8640 1 librados: setting wanted keys 2024-03-25T18:23:57.293 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.290+0000 7f70d01a8640 1 librados: calling monclient init 2024-03-25T18:23:57.293 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.290+0000 7f70d01a8640 10 monclient: init 2024-03-25T18:23:57.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.291+0000 7f70d01a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:57.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.291+0000 7f70d01a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:57.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.291+0000 7f70d01a8640 10 monclient(hunting): picked mon.noname-c con 0x7f70c80b1730 addr v1:172.21.15.63:6790/0 2024-03-25T18:23:57.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.291+0000 7f70d01a8640 10 monclient(hunting): picked mon.noname-b con 0x7f70c80b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T18:23:57.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.291+0000 7f70d01a8640 10 monclient(hunting): picked mon.noname-a con 0x7f70c814be90 addr v1:172.21.15.63:6789/0 2024-03-25T18:23:57.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.291+0000 7f70d01a8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c81403f0 con 0x7f70c814be90 2024-03-25T18:23:57.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.291+0000 7f70d01a8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c81408c0 con 0x7f70c80b4d60 2024-03-25T18:23:57.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.291+0000 7f70d01a8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c81410f0 con 0x7f70c80b1730 2024-03-25T18:23:57.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.291+0000 7f70d01a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:57.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.291+0000 7f70d01a8640 10 monclient(hunting): authenticate will time out at 8131.477649s 2024-03-25T18:23:57.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.303+0000 7f70ce745640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f70c814be90 0x7f70c812fd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:34448/0 (socket says 172.21.15.63:34448) 2024-03-25T18:23:57.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.303+0000 7f70ce745640 1 -- 172.21.15.63:0/2137604413 learned_addr learned my addr 172.21.15.63:0/2137604413 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:23:57.305 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.304+0000 7f70be7fc640 10 client.?.objecter ms_handle_connect 0x7f70c814be90 2024-03-25T18:23:57.305 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.304+0000 7f70be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:57.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.311+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340082267 0 0) 0x7f70c81403f0 con 0x7f70c814be90 2024-03-25T18:23:57.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.311+0000 7f70be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:57.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.311+0000 7f70be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:57.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.311+0000 7f70be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:57.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.311+0000 7f70be7fc640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:23:57.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.312+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70ac004eb0 con 0x7f70c814be90 2024-03-25T18:23:57.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.317+0000 7f70be7fc640 10 client.?.objecter ms_handle_connect 0x7f70c80b1730 2024-03-25T18:23:57.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.317+0000 7f70be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:57.320 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.319+0000 7f70be7fc640 10 client.?.objecter ms_handle_connect 0x7f70c80b4d60 2024-03-25T18:23:57.320 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.319+0000 7f70be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:57.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.324+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2240468348 0 0) 0x7f70ac004eb0 con 0x7f70c814be90 2024-03-25T18:23:57.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.324+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70c81403f0 con 0x7f70c814be90 2024-03-25T18:23:57.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.324+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f70c0001c90 con 0x7f70c814be90 2024-03-25T18:23:57.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.324+0000 7f70be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:57.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.324+0000 7f70be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:57.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.324+0000 7f70be7fc640 10 monclient(hunting): dump: 2024-03-25T18:23:57.326 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:57.326 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:57.326 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:57.326 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:57.326 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:57.326 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:57.326 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:57.326 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:57.326 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:57.326 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:57.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.324+0000 7f70be7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:57.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.329+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117505444 0 0) 0x7f70c81408c0 con 0x7f70c80b4d60 2024-03-25T18:23:57.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.329+0000 7f70be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:57.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.329+0000 7f70be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:57.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.329+0000 7f70be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:57.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.329+0000 7f70be7fc640 10 monclient(hunting): my global_id is 4750 2024-03-25T18:23:57.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.329+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70ac004eb0 con 0x7f70c80b4d60 2024-03-25T18:23:57.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.333+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164441462 0 0) 0x7f70c81410f0 con 0x7f70c80b1730 2024-03-25T18:23:57.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.333+0000 7f70be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:57.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.333+0000 7f70be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:57.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.333+0000 7f70be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:57.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.333+0000 7f70be7fc640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:23:57.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.333+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70c81408c0 con 0x7f70c80b1730 2024-03-25T18:23:57.337 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3672847065 0 0) 0x7f70c81403f0 con 0x7f70c814be90 2024-03-25T18:23:57.337 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 >> v1:172.21.15.63:6790/0 conn(0x7f70c80b1730 legacy=0x7f70c81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:57.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 >> v1:172.21.15.81:6789/0 conn(0x7f70c80b4d60 legacy=0x7f70c81d7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:57.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70be7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:57.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70be7fc640 1 monclient: found mon.a 2024-03-25T18:23:57.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:57.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70be7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:57.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c8143900 con 0x7f70c814be90 2024-03-25T18:23:57.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70be7fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:57.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70be7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:57.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:57.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70d01a8640 5 monclient: authenticate success, global_id 4707 2024-03-25T18:23:57.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70d01a8640 10 monclient: _renew_subs 2024-03-25T18:23:57.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70d01a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:57.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70d01a8640 1 -- 172.21.15.63:0/2137604413 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c8141de0 con 0x7f70c814be90 2024-03-25T18:23:57.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70d01a8640 10 client.4707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:57.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70d01a8640 10 monclient: _renew_subs 2024-03-25T18:23:57.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70d01a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:57.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70d01a8640 1 -- 172.21.15.63:0/2137604413 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c81d83d0 con 0x7f70c814be90 2024-03-25T18:23:57.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.336+0000 7f70d01a8640 1 librados: init done 2024-03-25T18:23:57.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.337+0000 7f70d01a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:57.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.337+0000 7f70d01a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:57.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.337+0000 7f70d01a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:57.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.337+0000 7f70d01a8640 1 -- 172.21.15.63:0/2137604413 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c81401f0 con 0x7f70c814be90 2024-03-25T18:23:57.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.668+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70c0003990 con 0x7f70c814be90 2024-03-25T18:23:57.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.668+0000 7f70be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:57.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.668+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f70c0003da0 con 0x7f70c814be90 2024-03-25T18:23:57.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.668+0000 7f70be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:57.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.668+0000 7f70be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:57.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.668+0000 7f70be7fc640 10 monclient: dump: 2024-03-25T18:23:57.669 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:23:57.670 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:23:57.670 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:57.670 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:23:57.670 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:57.670 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:23:57.670 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:23:57.670 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:23:57.670 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:23:57.670 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:23:57.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.668+0000 7f70bffff640 4 set_mon_vals no callback set 2024-03-25T18:23:57.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.668+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f70c0003f80 con 0x7f70c814be90 2024-03-25T18:23:57.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.669+0000 7f70cdf44640 10 monclient: get_auth_request con 0x7f70ac06de80 auth_method 0 2024-03-25T18:23:57.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.669+0000 7f70be7fc640 10 client.4707.objecter ms_handle_connect 0x7f70ac06de80 2024-03-25T18:23:57.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.675+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2076230003 0 0) 0x7f70c0054af0 con 0x7f70c814be90 2024-03-25T18:23:57.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.675+0000 7f70be7fc640 10 client.4707.objecter ms_dispatch 0x7f70c8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:23:57.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.675+0000 7f70be7fc640 3 client.4707.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:23:57.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.675+0000 7f70be7fc640 3 client.4707.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:23:57.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.675+0000 7f70be7fc640 20 client.4707.objecter dump_active .. 0 homeless 2024-03-25T18:23:57.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.675+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f70c0050600 con 0x7f70c814be90 2024-03-25T18:23:57.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.675+0000 7f70be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:57.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.675+0000 7f70be7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:57.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.905+0000 7f70d01a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:57.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.905+0000 7f70d01a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:57.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.905+0000 7f70d01a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T18:23:57.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.905+0000 7f70d01a8640 1 -- 172.21.15.63:0/2137604413 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70c80627b0 con 0x7f70c814be90 2024-03-25T18:23:57.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.945+0000 7f70be7fc640 1 -- 172.21.15.63:0/2137604413 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 515194931) 0x7f70c0052530 con 0x7f70c814be90 2024-03-25T18:23:57.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.945+0000 7f70be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:57.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.945+0000 7f70be7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:57.946 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:23:57.947 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":11,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:22:39.912010+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:22:39.912010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:57.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.947+0000 7f70d01a8640 10 librados: watch_flush enter 2024-03-25T18:23:57.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.947+0000 7f70d01a8640 10 librados: watch_flush exit 2024-03-25T18:23:57.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.947+0000 7f70d01a8640 20 client.4707.objecter shutdown clearing up homeless session... 2024-03-25T18:23:57.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.947+0000 7f70d01a8640 10 client.4707.objecter successfully canceled tick 2024-03-25T18:23:57.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.947+0000 7f70d01a8640 1 -- 172.21.15.63:0/2137604413 >> v1:172.21.15.81:6800/3640340873 conn(0x7f70ac06de80 legacy=0x7f70ac070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:57.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.947+0000 7f70d01a8640 10 monclient: shutdown 2024-03-25T18:23:57.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.947+0000 7f70d01a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:57.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.947+0000 7f70d01a8640 1 -- 172.21.15.63:0/2137604413 >> v1:172.21.15.63:6789/0 conn(0x7f70c814be90 legacy=0x7f70c812fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:57.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.947+0000 7f70d01a8640 1 -- 172.21.15.63:0/2137604413 shutdown_connections 2024-03-25T18:23:57.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.947+0000 7f70d01a8640 1 -- 172.21.15.63:0/2137604413 >> 172.21.15.63:0/2137604413 conn(0x7f70c80617a0 msgr2=0x7f70c813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:57.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.947+0000 7f70d01a8640 1 -- 172.21.15.63:0/2137604413 shutdown_connections 2024-03-25T18:23:57.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.947+0000 7f70d01a8640 1 -- 172.21.15.63:0/2137604413 wait complete. 2024-03-25T18:23:57.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:23:57.947+0000 7f70d01a8640 1 librados: shutdown 2024-03-25T18:23:57.959 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:04.764 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:04.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.879+0000 7f0bacc7b640 10 monclient: get_monmap_and_config 2024-03-25T18:24:04.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.879+0000 7f0bacc7b640 10 monclient: build_initial_monmap 2024-03-25T18:24:04.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.879+0000 7f0bacc7b640 10 monclient: monmap: 2024-03-25T18:24:04.880 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:24:04.880 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:24:04.880 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:24:04.880386+0000 2024-03-25T18:24:04.880 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:24:04.880386+0000 2024-03-25T18:24:04.880 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:04.881 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:24:04.881 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:24:04.881 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-25T18:24:04.881 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T18:24:04.881 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:24:04.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.880+0000 7f0bacc7b640 1 Processor -- start 2024-03-25T18:24:04.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.880+0000 7f0bacc7b640 1 -- start start 2024-03-25T18:24:04.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.880+0000 7f0bacc7b640 10 monclient: init 2024-03-25T18:24:04.882 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.880+0000 7f0bacc7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:04.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.880+0000 7f0bacc7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:04.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.880+0000 7f0bacc7b640 10 monclient(hunting): picked mon.noname-b con 0x7f0ba80a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T18:24:04.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.880+0000 7f0bacc7b640 10 monclient(hunting): picked mon.noname-a con 0x7f0ba813f710 addr v1:172.21.15.63:6789/0 2024-03-25T18:24:04.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.880+0000 7f0bacc7b640 10 monclient(hunting): picked mon.noname-c con 0x7f0ba81430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:24:04.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.880+0000 7f0bacc7b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba8142520 con 0x7f0ba813f710 2024-03-25T18:24:04.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.880+0000 7f0bacc7b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba80b28a0 con 0x7f0ba81430c0 2024-03-25T18:24:04.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.881+0000 7f0bacc7b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba80b2a30 con 0x7f0ba80a98f0 2024-03-25T18:24:04.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.881+0000 7f0bacc7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:04.883 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.881+0000 7f0bacc7b640 10 monclient(hunting): authenticate will time out at 8139.067361s 2024-03-25T18:24:04.896 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.895+0000 7f0ba659c640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f0ba80a98f0 0x7f0ba80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:58474/0 (socket says 172.21.15.63:58474) 2024-03-25T18:24:04.896 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.895+0000 7f0ba659c640 1 -- 172.21.15.63:0/1313026781 learned_addr learned my addr 172.21.15.63:0/1313026781 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:24:04.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.907+0000 7f0ba559a640 1 -- 172.21.15.63:0/1313026781 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10298367 0 0) 0x7f0ba80b2a30 con 0x7f0ba80a98f0 2024-03-25T18:24:04.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.907+0000 7f0ba559a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:04.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.907+0000 7f0ba559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:04.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.907+0000 7f0ba559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:04.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.907+0000 7f0ba559a640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:24:04.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.907+0000 7f0ba559a640 1 -- 172.21.15.63:0/1313026781 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b90003520 con 0x7f0ba80a98f0 2024-03-25T18:24:04.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.908+0000 7f0ba559a640 1 -- 172.21.15.63:0/1313026781 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269653551 0 0) 0x7f0ba80b28a0 con 0x7f0ba81430c0 2024-03-25T18:24:04.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.908+0000 7f0ba559a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:04.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.908+0000 7f0ba559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:04.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.908+0000 7f0ba559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:04.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.908+0000 7f0ba559a640 10 monclient(hunting): my global_id is 4753 2024-03-25T18:24:04.909 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.908+0000 7f0ba559a640 1 -- 172.21.15.63:0/1313026781 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba80b2a30 con 0x7f0ba81430c0 2024-03-25T18:24:04.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.915+0000 7f0ba559a640 1 -- 172.21.15.63:0/1313026781 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603622026 0 0) 0x7f0ba8142520 con 0x7f0ba813f710 2024-03-25T18:24:04.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.915+0000 7f0ba559a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:04.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.915+0000 7f0ba559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:04.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.915+0000 7f0ba559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:04.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.915+0000 7f0ba559a640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:24:04.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.915+0000 7f0ba559a640 1 -- 172.21.15.63:0/1313026781 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba80b28a0 con 0x7f0ba813f710 2024-03-25T18:24:04.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0ba559a640 1 -- 172.21.15.63:0/1313026781 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 786405873 0 0) 0x7f0ba80b2a30 con 0x7f0ba81430c0 2024-03-25T18:24:04.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0ba559a640 1 -- 172.21.15.63:0/1313026781 >> v1:172.21.15.63:6790/0 conn(0x7f0ba80a98f0 legacy=0x7f0ba80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:04.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0ba559a640 1 -- 172.21.15.63:0/1313026781 >> v1:172.21.15.63:6789/0 conn(0x7f0ba813f710 legacy=0x7f0ba81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:04.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0ba559a640 10 monclient: _finish_hunting 0 2024-03-25T18:24:04.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0ba559a640 1 monclient: found mon.noname-c 2024-03-25T18:24:04.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0ba559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:04.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0ba559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-25T18:24:04.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0ba559a640 1 -- 172.21.15.63:0/1313026781 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba80b44c0 con 0x7f0ba81430c0 2024-03-25T18:24:04.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0ba559a640 10 monclient: _finish_auth 0 2024-03-25T18:24:04.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0ba559a640 10 monclient: _check_auth_tickets 2024-03-25T18:24:04.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0ba559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:04.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0ba559a640 1 -- 172.21.15.63:0/1313026781 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f0ba00019f0 con 0x7f0ba81430c0 2024-03-25T18:24:04.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0ba559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:04.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0ba559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:04.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0ba559a640 10 monclient: dump: 2024-03-25T18:24:04.924 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:24:04.924 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:24:04.924 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:24:04.924 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:24:04.924 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:04.924 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:24:04.924 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:24:04.925 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:24:04.925 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:24:04.925 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:24:04.925 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0bacc7b640 5 monclient: authenticate success, global_id 4753 2024-03-25T18:24:04.925 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.922+0000 7f0bacc7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:04.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.934+0000 7f0ba559a640 1 -- 172.21.15.63:0/1313026781 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ba0001bd0 con 0x7f0ba81430c0 2024-03-25T18:24:04.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.934+0000 7f0ba559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:04.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.934+0000 7f0ba559a640 1 -- 172.21.15.63:0/1313026781 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f0ba0001fe0 con 0x7f0ba81430c0 2024-03-25T18:24:04.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.934+0000 7f0bacc7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:04.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.934+0000 7f0bacc7b640 4 set_mon_vals no callback set 2024-03-25T18:24:04.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.934+0000 7f0ba559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:04.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.934+0000 7f0ba559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:04.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.934+0000 7f0ba559a640 10 monclient: dump: 2024-03-25T18:24:04.936 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:24:04.936 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:24:04.936 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:24:04.936 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:24:04.936 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:04.936 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:24:04.936 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:24:04.936 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:24:04.936 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:24:04.936 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:24:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.935+0000 7f0bacc7b640 10 monclient: shutdown 2024-03-25T18:24:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.935+0000 7f0bacc7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.935+0000 7f0bacc7b640 1 -- 172.21.15.63:0/1313026781 >> v1:172.21.15.81:6789/0 conn(0x7f0ba81430c0 legacy=0x7f0ba814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.935+0000 7f0bacc7b640 1 -- 172.21.15.63:0/1313026781 shutdown_connections 2024-03-25T18:24:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.935+0000 7f0bacc7b640 1 -- 172.21.15.63:0/1313026781 >> 172.21.15.63:0/1313026781 conn(0x7f0ba80b7ea0 msgr2=0x7f0ba80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.936+0000 7f0bacc7b640 1 -- 172.21.15.63:0/1313026781 shutdown_connections 2024-03-25T18:24:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.936+0000 7f0bacc7b640 1 -- 172.21.15.63:0/1313026781 wait complete. 2024-03-25T18:24:04.938 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.937+0000 7f0bacc7b640 10 monclient: build_initial_monmap 2024-03-25T18:24:04.938 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.937+0000 7f0bacc7b640 10 monclient: monmap: 2024-03-25T18:24:04.939 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T18:24:04.939 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:04.939 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T18:24:04.939 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T18:24:04.939 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:04.939 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T18:24:04.939 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T18:24:04.939 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-25T18:24:04.939 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T18:24:04.939 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:24:04.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.937+0000 7f0bacc7b640 1 librados: starting msgr at 2024-03-25T18:24:04.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.937+0000 7f0bacc7b640 1 librados: starting objecter 2024-03-25T18:24:04.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.937+0000 7f0bacc7b640 1 Processor -- start 2024-03-25T18:24:04.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.937+0000 7f0bacc7b640 1 -- start start 2024-03-25T18:24:04.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.937+0000 7f0bacc7b640 1 librados: setting wanted keys 2024-03-25T18:24:04.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.937+0000 7f0bacc7b640 1 librados: calling monclient init 2024-03-25T18:24:04.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.937+0000 7f0bacc7b640 10 monclient: init 2024-03-25T18:24:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.938+0000 7f0bacc7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.938+0000 7f0bacc7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.938+0000 7f0bacc7b640 10 monclient(hunting): picked mon.noname-a con 0x7f0ba80a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T18:24:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.938+0000 7f0bacc7b640 10 monclient(hunting): picked mon.noname-c con 0x7f0ba813f710 addr v1:172.21.15.63:6790/0 2024-03-25T18:24:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.938+0000 7f0bacc7b640 10 monclient(hunting): picked mon.noname-b con 0x7f0ba81430c0 addr v1:172.21.15.81:6789/0 2024-03-25T18:24:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.938+0000 7f0bacc7b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba81d7590 con 0x7f0ba80a98f0 2024-03-25T18:24:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.938+0000 7f0bacc7b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba81d51f0 con 0x7f0ba81430c0 2024-03-25T18:24:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.938+0000 7f0bacc7b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba81d7830 con 0x7f0ba813f710 2024-03-25T18:24:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.938+0000 7f0bacc7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.938+0000 7f0bacc7b640 10 monclient(hunting): authenticate will time out at 8139.124763s 2024-03-25T18:24:04.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.958+0000 7f0ba5d9b640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f0ba813f710 0x7f0ba81d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:58524/0 (socket says 172.21.15.63:58524) 2024-03-25T18:24:04.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.958+0000 7f0ba5d9b640 1 -- 172.21.15.63:0/2475204721 learned_addr learned my addr 172.21.15.63:0/2475204721 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T18:24:04.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.958+0000 7f0ba6d9d640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f0ba81430c0 0x7f0ba81e1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.63:46690/0 (socket says 172.21.15.63:46690) 2024-03-25T18:24:04.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.959+0000 7f0b96ffd640 10 client.?.objecter ms_handle_connect 0x7f0ba81430c0 2024-03-25T18:24:04.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.959+0000 7f0b96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:04.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.959+0000 7f0b96ffd640 10 client.?.objecter ms_handle_connect 0x7f0ba813f710 2024-03-25T18:24:04.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.959+0000 7f0b96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:04.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.967+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929572156 0 0) 0x7f0ba81d51f0 con 0x7f0ba81430c0 2024-03-25T18:24:04.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.967+0000 7f0b96ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:04.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.967+0000 7f0b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:04.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.967+0000 7f0b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:04.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.967+0000 7f0b96ffd640 10 monclient(hunting): my global_id is 4756 2024-03-25T18:24:04.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.967+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b88004eb0 con 0x7f0ba81430c0 2024-03-25T18:24:04.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.968+0000 7f0b96ffd640 10 client.?.objecter ms_handle_connect 0x7f0ba80a98f0 2024-03-25T18:24:04.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.968+0000 7f0b96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:04.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.968+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261526719 0 0) 0x7f0ba81d7830 con 0x7f0ba813f710 2024-03-25T18:24:04.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.968+0000 7f0b96ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:04.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.968+0000 7f0b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:04.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.968+0000 7f0b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:04.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.968+0000 7f0b96ffd640 10 monclient(hunting): my global_id is 4730 2024-03-25T18:24:04.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.968+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba81d51f0 con 0x7f0ba813f710 2024-03-25T18:24:04.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.970+0000 7f0ba659c640 1 -- 172.21.15.63:0/2475204721 >> v1:172.21.15.63:6789/0 conn(0x7f0ba80a98f0 legacy=0x7f0ba8137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T18:24:04.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.970+0000 7f0ba659c640 1 -- 172.21.15.63:0/2475204721 >> v1:172.21.15.63:6789/0 conn(0x7f0ba80a98f0 legacy=0x7f0ba8137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:24:04.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.970+0000 7f0ba659c640 1 --1- 172.21.15.63:0/2475204721 >> v1:172.21.15.63:6789/0 conn(0x7f0ba80a98f0 0x7f0ba8137af0 :-1 s=OPENED pgs=87018 cs=1 l=1).handle_message read tag failed 2024-03-25T18:24:04.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.970+0000 7f0ba659c640 1 --1- 172.21.15.63:0/2475204721 >> v1:172.21.15.63:6789/0 conn(0x7f0ba80a98f0 0x7f0ba8137af0 :-1 s=OPENED pgs=87018 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:24:04.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.970+0000 7f0b96ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.63:6789/0 2024-03-25T18:24:04.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.975+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1480577894 0 0) 0x7f0b88004eb0 con 0x7f0ba81430c0 2024-03-25T18:24:04.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.975+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ba81d7830 con 0x7f0ba81430c0 2024-03-25T18:24:04.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.975+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f0ba0001f20 con 0x7f0ba81430c0 2024-03-25T18:24:04.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.975+0000 7f0b96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:04.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.975+0000 7f0b96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:04.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.975+0000 7f0b96ffd640 10 monclient(hunting): dump: 2024-03-25T18:24:04.976 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:24:04.976 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:24:04.977 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:24:04.977 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:24:04.977 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:04.977 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:24:04.977 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:24:04.977 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:24:04.977 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:24:04.977 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:24:04.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.975+0000 7f0b96ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:04.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.991+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3764668782 0 0) 0x7f0ba81d51f0 con 0x7f0ba813f710 2024-03-25T18:24:04.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.991+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b88004eb0 con 0x7f0ba813f710 2024-03-25T18:24:04.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.991+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f0b9c001970 con 0x7f0ba813f710 2024-03-25T18:24:04.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.991+0000 7f0b96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:04.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.991+0000 7f0b96ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:04.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.991+0000 7f0b96ffd640 10 monclient(hunting): dump: 2024-03-25T18:24:04.993 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:24:04.993 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:24:04.993 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:24:04.993 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:24:04.993 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:04.993 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:24:04.993 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:24:04.993 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:24:04.993 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:24:04.993 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:24:04.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.991+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3811768791 0 0) 0x7f0ba81d7830 con 0x7f0ba81430c0 2024-03-25T18:24:04.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.991+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 >> v1:172.21.15.63:6790/0 conn(0x7f0ba813f710 legacy=0x7f0ba81d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.991+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 >> v1:172.21.15.63:6789/0 conn(0x7f0ba80a98f0 legacy=0x7f0ba8137af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:24:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.991+0000 7f0b96ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.991+0000 7f0b96ffd640 1 monclient: found mon.b 2024-03-25T18:24:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.992+0000 7f0b96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.992+0000 7f0b96ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:24:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.992+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba81d8440 con 0x7f0ba81430c0 2024-03-25T18:24:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.992+0000 7f0b96ffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.992+0000 7f0b96ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.992+0000 7f0b96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.992+0000 7f0bacc7b640 5 monclient: authenticate success, global_id 4756 2024-03-25T18:24:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.992+0000 7f0bacc7b640 10 monclient: _renew_subs 2024-03-25T18:24:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.992+0000 7f0bacc7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:24:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.992+0000 7f0bacc7b640 1 -- 172.21.15.63:0/2475204721 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba8138800 con 0x7f0ba81430c0 2024-03-25T18:24:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.992+0000 7f0bacc7b640 10 client.4756.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.992+0000 7f0bacc7b640 10 monclient: _renew_subs 2024-03-25T18:24:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.992+0000 7f0bacc7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:24:04.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.992+0000 7f0bacc7b640 1 -- 172.21.15.63:0/2475204721 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba8138c90 con 0x7f0ba81430c0 2024-03-25T18:24:04.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.992+0000 7f0bacc7b640 1 librados: init done 2024-03-25T18:24:04.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.993+0000 7f0bacc7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:04.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.993+0000 7f0bacc7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:04.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.993+0000 7f0bacc7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:24:04.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:04.993+0000 7f0bacc7b640 1 -- 172.21.15.63:0/2475204721 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba80abc70 con 0x7f0ba81430c0 2024-03-25T18:24:05.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.256+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ba0003c20 con 0x7f0ba81430c0 2024-03-25T18:24:05.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.256+0000 7f0b96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:05.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.256+0000 7f0ba4d99640 4 set_mon_vals no callback set 2024-03-25T18:24:05.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.266+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2147266282 0 0) 0x7f0ba00015b0 con 0x7f0ba81430c0 2024-03-25T18:24:05.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.266+0000 7f0b96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:05.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.266+0000 7f0b96ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:05.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.266+0000 7f0b96ffd640 10 monclient: dump: 2024-03-25T18:24:05.267 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T18:24:05.267 INFO:teuthology.misc.health.smithi063.stderr:fsid 9f4ce699-6031-4b74-90db-a99103627708 2024-03-25T18:24:05.267 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T18:11:45.337036+0000 2024-03-25T18:24:05.267 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T18:11:45.337036+0000 2024-03-25T18:24:05.267 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:05.267 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T18:24:05.267 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T18:24:05.268 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-25T18:24:05.268 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T18:24:05.268 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T18:24:05.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.266+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2953147110 0 0) 0x7f0ba0004070 con 0x7f0ba81430c0 2024-03-25T18:24:05.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.267+0000 7f0ba659c640 10 monclient: get_auth_request con 0x7f0b8806de40 auth_method 0 2024-03-25T18:24:05.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.268+0000 7f0b96ffd640 10 client.4756.objecter ms_handle_connect 0x7f0b8806de40 2024-03-25T18:24:05.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.272+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2076230003 0 0) 0x7f0ba0054b90 con 0x7f0ba81430c0 2024-03-25T18:24:05.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.272+0000 7f0b96ffd640 10 client.4756.objecter ms_dispatch 0x7f0ba8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:05.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.272+0000 7f0b96ffd640 3 client.4756.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:05.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.272+0000 7f0b96ffd640 3 client.4756.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:05.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.272+0000 7f0b96ffd640 20 client.4756.objecter dump_active .. 0 homeless 2024-03-25T18:24:05.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.272+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ba00504d0 con 0x7f0ba81430c0 2024-03-25T18:24:05.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.272+0000 7f0b96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:05.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.272+0000 7f0b96ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:05.332 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:47.024 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:05.387 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:47.080 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:05.430 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:11:47.091 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:05.431 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:47.123 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:05.439 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:11:47.100 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:05.474 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:47.166 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:05.477 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:47.169 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:05.479 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:11:47.139 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:05.506 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:47.198 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:05.519 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.517+0000 7f0bacc7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:05.519 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.517+0000 7f0bacc7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:05.519 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.517+0000 7f0bacc7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-25T18:24:05.519 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.517+0000 7f0bacc7b640 1 -- 172.21.15.63:0/2475204721 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ba80627b0 con 0x7f0ba81430c0 2024-03-25T18:24:05.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.548+0000 7f0b96ffd640 1 -- 172.21.15.63:0/2475204721 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 515194931) 0x7f0ba00525d0 con 0x7f0ba81430c0 2024-03-25T18:24:05.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.548+0000 7f0b96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:05.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.548+0000 7f0b96ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:05.550 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T18:24:05.550 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":11,"fsid":"9f4ce699-6031-4b74-90db-a99103627708","created":"2024-03-25T18:12:35.671678+0000","modified":"2024-03-25T18:22:39.912010+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:22:39.912010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7d4153f5-c1a1-4a43-9781-52018a420e7e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6d5c570a-fd90-450a-a56c-f646b4d63b2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"cdfc79b2-cb55-4ddb-bfac-682073ac86dc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d53bda7b-3db2-454a-a647-50c86732edcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0b711e9-7872-4700-83a6-986664c998d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9c189d47-3570-4000-92d2-c1a733788e62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b8938e4-7c72-453d-bed9-df8bce519abb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1c628d65-bba1-4a42-bc80-dc5197e22e2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:05.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.551+0000 7f0bacc7b640 10 librados: watch_flush enter 2024-03-25T18:24:05.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.551+0000 7f0bacc7b640 10 librados: watch_flush exit 2024-03-25T18:24:05.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.551+0000 7f0bacc7b640 20 client.4756.objecter shutdown clearing up homeless session... 2024-03-25T18:24:05.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.551+0000 7f0bacc7b640 10 client.4756.objecter successfully canceled tick 2024-03-25T18:24:05.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.551+0000 7f0bacc7b640 1 -- 172.21.15.63:0/2475204721 >> v1:172.21.15.81:6800/3640340873 conn(0x7f0b8806de40 legacy=0x7f0b88070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:05.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.551+0000 7f0bacc7b640 10 monclient: shutdown 2024-03-25T18:24:05.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.551+0000 7f0bacc7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:05.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.551+0000 7f0bacc7b640 1 -- 172.21.15.63:0/2475204721 >> v1:172.21.15.81:6789/0 conn(0x7f0ba81430c0 legacy=0x7f0ba81e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:05.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.551+0000 7f0bacc7b640 1 -- 172.21.15.63:0/2475204721 shutdown_connections 2024-03-25T18:24:05.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.551+0000 7f0bacc7b640 1 -- 172.21.15.63:0/2475204721 >> 172.21.15.63:0/2475204721 conn(0x7f0ba80b7ea0 msgr2=0x7f0ba8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:05.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.551+0000 7f0bacc7b640 1 -- 172.21.15.63:0/2475204721 shutdown_connections 2024-03-25T18:24:05.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.551+0000 7f0bacc7b640 1 -- 172.21.15.63:0/2475204721 wait complete. 2024-03-25T18:24:05.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T18:24:05.551+0000 7f0bacc7b640 1 librados: shutdown 2024-03-25T18:24:05.562 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:05.563 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:24:05.564 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:24:05.564 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:24:05.564 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:24:07.608 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:11:25.358 35499== 2024-03-25T18:24:07.608 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:11:25.358 35499== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:24:07.646 INFO:tasks.ceph.osd.0.smithi063.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:24:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:24:11.666 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi063 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:24:11.667 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:24:11.667 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:24:11.667 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:24:13.506 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:11:31.263 35498== 2024-03-25T18:24:13.506 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:11:31.263 35498== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:24:13.547 INFO:tasks.ceph.osd.1.smithi063.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:24:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:24:17.769 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi063 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:24:17.770 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:24:17.770 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:24:17.770 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:24:19.733 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:11:37.475 35500== 2024-03-25T18:24:19.734 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:11:37.475 35500== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:24:19.774 INFO:tasks.ceph.osd.2.smithi063.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:24:23.871 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:24:23.872 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi063 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:24:23.872 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:24:23.872 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:24:23.873 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:24:25.345 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:12:07.037 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:25.361 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:12:07.053 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:25.397 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:12:07.089 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:25.404 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:12:07.096 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:25.414 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:12:07.107 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:25.423 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:12:07.083 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:25.433 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:12:07.093 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:25.447 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:12:07.139 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:25.458 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:12:07.118 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:25.590 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:11:43.330 35501== 2024-03-25T18:24:25.590 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:11:43.330 35501== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:24:25.626 INFO:tasks.ceph.osd.3.smithi063.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:24:29.974 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:24:29.974 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi063 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:24:29.974 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:24:29.975 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:24:29.975 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:24:31.887 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:11:49.647 35478== 2024-03-25T18:24:31.888 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:11:49.647 35478== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:24:31.921 INFO:tasks.ceph.osd.4.smithi081.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:24:36.076 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:24:36.077 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:24:36.077 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:24:36.077 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:24:36.078 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:24:37.904 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:11:55.664 35479== 2024-03-25T18:24:37.904 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:11:55.664 35479== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:24:37.940 INFO:tasks.ceph.osd.5.smithi081.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:24:42.179 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:24:42.180 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:24:42.180 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:24:42.180 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:24:42.180 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:24:43.978 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:12:01.734 35480== 2024-03-25T18:24:43.978 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:12:01.734 35480== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:24:44.021 INFO:tasks.ceph.osd.6.smithi081.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:24:45.421 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:12:27.113 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:45.427 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:12:27.119 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:45.440 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:12:27.132 35184== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:45.467 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:12:27.128 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:45.472 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:12:27.164 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:45.473 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:12:27.133 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:45.473 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:12:27.166 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:45.485 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:12:27.146 34854== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:45.489 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:12:27.182 35187== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:48.282 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:24:48.282 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:24:48.283 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:24:48.283 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:24:48.283 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:24:49.823 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:12:07.578 35481== 2024-03-25T18:24:49.823 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:12:07.578 35481== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:24:49.853 INFO:tasks.ceph.osd.7.smithi081.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:24:54.385 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:24:54.385 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:24:54.385 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:24:54.385 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:24:54.386 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:24:54.386 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:24:54.411 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:24:54.412 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:24:54.412 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:24:54.440 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:24:54.440 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:24:54.440 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:24:54.440 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:24:54.467 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-03-25T18:24:54.456+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35153) UID: 0 2024-03-25T18:24:54.468 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-03-25T18:24:54.458+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:26:27.834 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:14:09.525 35184== 2024-03-25T18:26:27.834 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:14:09.525 35184== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:27.843 INFO:tasks.ceph.mon.a.smithi063.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:32.062 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:32.063 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi063 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:26:32.063 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:26:32.064 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:26:32.064 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:32.080 INFO:tasks.ceph.mon.c.smithi063.stderr:2024-03-25T18:26:32.068+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35155) UID: 0 2024-03-25T18:26:32.081 INFO:tasks.ceph.mon.c.smithi063.stderr:2024-03-25T18:26:32.070+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:26:32.606 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:14:14.298 35187== 2024-03-25T18:26:32.607 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:14:14.298 35187== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:32.615 INFO:tasks.ceph.mon.c.smithi063.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:38.166 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:38.166 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi063 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:26:38.167 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:26:38.167 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:26:38.167 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:38.183 INFO:tasks.ceph.mon.b.smithi081.stderr:2024-03-25T18:26:38.171+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34850) UID: 0 2024-03-25T18:26:38.184 INFO:tasks.ceph.mon.b.smithi081.stderr:2024-03-25T18:26:38.172+0000 9f86640 -1 mon.b@1(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:26:38.684 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:14:20.344 34854== 2024-03-25T18:26:38.684 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:14:20.344 34854== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:38.693 INFO:tasks.ceph.mon.b.smithi081.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:44.269 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:44.270 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:26:44.270 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:26:44.270 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:26:44.271 DEBUG:teuthology.orchestra.run.smithi063:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:26:44.319 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T18:26:44.320 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:26:44.418 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T18:26:44.418 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:26:44.609 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T18:26:44.609 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:26:44.677 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T18:26:44.678 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:26:44.743 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T18:26:44.744 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:26:44.857 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T18:26:44.858 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:26:44.905 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T18:26:44.905 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:26:44.964 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T18:26:44.965 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:26:45.050 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:26:45.052 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256/data/mon.a.tgz 2024-03-25T18:26:45.052 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-03-25T18:26:45.068 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.4DJhFOgW1F 2024-03-25T18:26:45.068 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.4DJhFOgW1F 2024-03-25T18:26:45.307 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.4DJhFOgW1F 2024-03-25T18:26:45.395 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.4DJhFOgW1F is 133KB 2024-03-25T18:26:45.446 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.4DJhFOgW1F 2024-03-25T18:26:45.461 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256/data/mon.c.tgz 2024-03-25T18:26:45.461 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-03-25T18:26:45.515 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.Ew1HkdhG4c 2024-03-25T18:26:45.516 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Ew1HkdhG4c 2024-03-25T18:26:45.609 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.Ew1HkdhG4c 2024-03-25T18:26:45.694 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.Ew1HkdhG4c is 164KB 2024-03-25T18:26:45.746 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.Ew1HkdhG4c 2024-03-25T18:26:45.761 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256/data/mon.b.tgz 2024-03-25T18:26:45.761 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-03-25T18:26:45.779 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.QZya5QDtvS 2024-03-25T18:26:45.779 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.QZya5QDtvS 2024-03-25T18:26:45.881 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.QZya5QDtvS 2024-03-25T18:26:45.969 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.QZya5QDtvS is 155KB 2024-03-25T18:26:46.022 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.QZya5QDtvS 2024-03-25T18:26:46.037 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:26:46.038 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:26:46.040 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:26:46.428 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:26:46.705 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:26:46.707 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256/remote/smithi063/crash 2024-03-25T18:26:46.708 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:26:46.736 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256/remote/smithi081/crash 2024-03-25T18:26:46.737 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:26:46.765 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:26:46.765 DEBUG:teuthology.orchestra.run.smithi063:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:26:46.780 DEBUG:teuthology.orchestra.run.smithi081:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:26:46.854 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:26:46.855 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:26:46.855 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:26:46.855 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:26:46.855 INFO:teuthology.orchestra.run.smithi081.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:26:46.855 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:26:46.855 INFO:teuthology.orchestra.run.smithi081.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:26:46.855 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:26:46.855 INFO:teuthology.orchestra.run.smithi081.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:26:46.855 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35009.log 2024-03-25T18:26:46.855 INFO:teuthology.orchestra.run.smithi081.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:26:46.855 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:26:46.855 INFO:teuthology.orchestra.run.smithi081.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:26:46.855 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35169.log 2024-03-25T18:26:46.855 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35129.log 2024-03-25T18:26:46.856 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35009.log.gz 2024-03-25T18:26:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35169.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:26:46.856 INFO:teuthology.orchestra.run.smithi081.stderr: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:26:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35129.log: gzip 87.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.35169.log.gz --verbose 2024-03-25T18:26:46.856 INFO:teuthology.orchestra.run.smithi081.stderr: -- /var/log/ceph/ceph-client.admin.35049.log 2024-03-25T18:26:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.4.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35129.log.gzgzip 2024-03-25T18:26:46.856 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:26:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35049.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:26:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.7.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35049.log.gz 2024-03-25T18:26:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35089.log 2024-03-25T18:26:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:26:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.35089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35260.log 2024-03-25T18:26:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:26:46.856 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35089.log.gz 2024-03-25T18:26:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35260.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:26:46.857 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35260.log.gz 2024-03-25T18:26:46.857 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35220.log 2024-03-25T18:26:46.857 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35375.log 2024-03-25T18:26:46.857 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35220.log: /var/log/ceph/ceph-mon.b.log: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:26:46.857 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35220.log.gz 2024-03-25T18:26:46.857 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35375.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.35375.log.gz 2024-03-25T18:26:46.860 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:26:46.860 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35195.log 2024-03-25T18:26:46.860 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5/var/log/ceph/ceph-mon.a.log: --verbose -- /var/log/ceph/ceph-client.admin.40127.log 2024-03-25T18:26:46.860 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35682.log 2024-03-25T18:26:46.860 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40831.log 2024-03-25T18:26:46.860 INFO:teuthology.orchestra.run.smithi063.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35195.log.gz 2024-03-25T18:26:46.860 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39815.log 2024-03-25T18:26:46.860 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35682.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40127.log.gz 2024-03-25T18:26:46.860 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40245.log 2024-03-25T18:26:46.860 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39893.log 2024-03-25T18:26:46.860 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35682.log.gz 2024-03-25T18:26:46.860 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40831.log.gz 2024-03-25T18:26:46.861 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41182.log 2024-03-25T18:26:46.861 INFO:teuthology.orchestra.run.smithi063.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39815.log.gz 2024-03-25T18:26:46.861 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40362.log 2024-03-25T18:26:46.861 INFO:teuthology.orchestra.run.smithi063.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40245.log.gz 2024-03-25T18:26:46.861 INFO:teuthology.orchestra.run.smithi063.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39893.log.gz 2024-03-25T18:26:46.861 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40987.log 2024-03-25T18:26:46.861 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41182.log: /var/log/ceph/ceph-client.admin.40362.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:26:46.861 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40987.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:26:46.861 INFO:teuthology.orchestra.run.smithi063.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41182.log.gz 2024-03-25T18:26:46.861 INFO:teuthology.orchestra.run.smithi063.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40362.log.gz 2024-03-25T18:26:46.861 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:26:46.861 INFO:teuthology.orchestra.run.smithi063.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40987.log.gz 2024-03-25T18:26:46.861 INFO:teuthology.orchestra.run.smithi063.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:26:46.861 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:26:46.862 INFO:teuthology.orchestra.run.smithi063.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:26:46.862 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:26:46.862 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/valgrind/osd.0.log: /var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:26:46.862 INFO:teuthology.orchestra.run.smithi063.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:26:46.862 INFO:teuthology.orchestra.run.smithi063.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:26:46.862 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41380.log 2024-03-25T18:26:46.862 INFO:teuthology.orchestra.run.smithi063.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:26:46.862 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41104.log 2024-03-25T18:26:46.862 INFO:teuthology.orchestra.run.smithi063.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:26:46.863 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41380.log: -- /var/log/ceph/ceph-client.admin.39190.log 2024-03-25T18:26:46.863 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40870.log 2024-03-25T18:26:46.863 INFO:teuthology.orchestra.run.smithi063.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41380.log.gz 2024-03-25T18:26:46.863 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41928.log 2024-03-25T18:26:46.863 INFO:teuthology.orchestra.run.smithi063.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41104.log.gz 2024-03-25T18:26:46.863 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40870.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39190.log.gz 2024-03-25T18:26:46.863 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40479.log 2024-03-25T18:26:46.863 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41928.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40870.log.gz 2024-03-25T18:26:46.863 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39659.log 2024-03-25T18:26:46.863 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37647.log 2024-03-25T18:26:46.863 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41928.log.gz 2024-03-25T18:26:46.864 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41460.log 2024-03-25T18:26:46.864 INFO:teuthology.orchestra.run.smithi063.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40479.log.gz 2024-03-25T18:26:46.864 INFO:teuthology.orchestra.run.smithi063.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39659.log.gz 2024-03-25T18:26:46.864 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41538.log 2024-03-25T18:26:46.864 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.37647.log: /var/log/ceph/ceph-client.admin.41460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41811.log 2024-03-25T18:26:46.864 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41694.log 2024-03-25T18:26:46.864 INFO:teuthology.orchestra.run.smithi063.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37647.log.gz 2024-03-25T18:26:46.864 INFO:teuthology.orchestra.run.smithi063.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41460.log.gz 2024-03-25T18:26:46.864 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40049.log 2024-03-25T18:26:46.864 INFO:teuthology.orchestra.run.smithi063.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41538.log.gz 2024-03-25T18:26:46.864 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41694.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41811.log.gz 2024-03-25T18:26:46.864 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40909.log 2024-03-25T18:26:46.864 INFO:teuthology.orchestra.run.smithi063.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41694.log.gz 2024-03-25T18:26:46.864 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39070.log 2024-03-25T18:26:46.865 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40049.log: /var/log/ceph/ceph-client.admin.40909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42201.log 2024-03-25T18:26:46.865 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39070.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40049.log.gz 2024-03-25T18:26:46.865 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39424.log 2024-03-25T18:26:46.865 INFO:teuthology.orchestra.run.smithi063.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40909.log.gz 2024-03-25T18:26:46.865 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.42201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42123.log 2024-03-25T18:26:46.865 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39070.log.gz 2024-03-25T18:26:46.865 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38821.log 2024-03-25T18:26:46.865 INFO:teuthology.orchestra.run.smithi063.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42201.log.gz 2024-03-25T18:26:46.865 INFO:teuthology.orchestra.run.smithi063.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39424.log.gz 2024-03-25T18:26:46.865 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:26:46.865 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.42123.log: /var/log/ceph/ceph-client.admin.38821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41263.log 2024-03-25T18:26:46.865 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph.audit.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42123.log.gz 2024-03-25T18:26:46.865 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39932.log 2024-03-25T18:26:46.865 INFO:teuthology.orchestra.run.smithi063.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38821.log.gz 2024-03-25T18:26:46.866 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41341.log 2024-03-25T18:26:46.866 INFO:teuthology.orchestra.run.smithi063.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:26:46.866 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40401.log 2024-03-25T18:26:46.866 INFO:teuthology.orchestra.run.smithi063.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41263.log.gz 2024-03-25T18:26:46.866 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42084.log 2024-03-25T18:26:46.866 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41341.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39932.log.gz 2024-03-25T18:26:46.866 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41577.log 2024-03-25T18:26:46.866 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.42084.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41341.log.gz 2024-03-25T18:26:46.866 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39854.log 2024-03-25T18:26:46.866 INFO:teuthology.orchestra.run.smithi063.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40401.log.gz 2024-03-25T18:26:46.866 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39463.log 2024-03-25T18:26:46.866 INFO:teuthology.orchestra.run.smithi063.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42084.log.gz 2024-03-25T18:26:46.866 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35524.log 2024-03-25T18:26:46.866 INFO:teuthology.orchestra.run.smithi063.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41577.log.gz 2024-03-25T18:26:46.866 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40010.log 2024-03-25T18:26:46.867 INFO:teuthology.orchestra.run.smithi063.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39854.log.gz 2024-03-25T18:26:46.867 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35524.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39463.log.gz 2024-03-25T18:26:46.867 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41143.log 2024-03-25T18:26:46.867 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42162.log 2024-03-25T18:26:46.867 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35524.log.gz 2024-03-25T18:26:46.867 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41143.log: 86.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.40010.log.gz 2024-03-25T18:26:46.867 INFO:teuthology.orchestra.run.smithi063.stderr: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29343.log 2024-03-25T18:26:46.867 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.42162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40558.log 2024-03-25T18:26:46.867 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41143.log.gz 2024-03-25T18:26:46.867 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph.tmp-client.admin.29343.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29343.log.gz 2024-03-25T18:26:46.867 INFO:teuthology.orchestra.run.smithi063.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42162.log.gz 2024-03-25T18:26:46.867 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41065.log 2024-03-25T18:26:46.867 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40167.log 2024-03-25T18:26:46.867 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40440.log 2024-03-25T18:26:46.868 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40558.log.gz 2024-03-25T18:26:46.868 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41616.log 2024-03-25T18:26:46.868 INFO:teuthology.orchestra.run.smithi063.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41065.log.gz 2024-03-25T18:26:46.868 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40714.log 2024-03-25T18:26:46.868 INFO:teuthology.orchestra.run.smithi063.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40167.log.gz 2024-03-25T18:26:46.868 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41616.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:26:46.868 INFO:teuthology.orchestra.run.smithi063.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40440.log.gz 2024-03-25T18:26:46.868 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35497.log 2024-03-25T18:26:46.868 INFO:teuthology.orchestra.run.smithi063.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41616.log.gz 2024-03-25T18:26:46.868 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-osd.1.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40714.log.gz 2024-03-25T18:26:46.868 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42045.log 2024-03-25T18:26:46.868 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39698.log 2024-03-25T18:26:46.868 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.42045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41967.log 2024-03-25T18:26:46.868 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35497.log.gz 2024-03-25T18:26:46.869 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39698.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.42045.log.gz 2024-03-25T18:26:46.869 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40675.log 2024-03-25T18:26:46.869 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41967.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39698.log.gz 2024-03-25T18:26:46.869 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39776.log 2024-03-25T18:26:46.869 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41499.log 2024-03-25T18:26:46.869 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40675.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41967.log.gz 2024-03-25T18:26:46.869 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36578.log 2024-03-25T18:26:46.869 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41499.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40675.log.gz 2024-03-25T18:26:46.869 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40636.log 2024-03-25T18:26:46.869 INFO:teuthology.orchestra.run.smithi063.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39776.log.gz 2024-03-25T18:26:46.869 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.36578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41850.log 2024-03-25T18:26:46.869 INFO:teuthology.orchestra.run.smithi063.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41499.log.gz 2024-03-25T18:26:46.869 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40636.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36578.log.gz 2024-03-25T18:26:46.869 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:26:46.869 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40753.log 2024-03-25T18:26:46.870 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40636.log.gz 2024-03-25T18:26:46.870 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41889.log 2024-03-25T18:26:46.870 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41850.log.gz 2024-03-25T18:26:46.870 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42006.log 2024-03-25T18:26:46.870 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39542.log 2024-03-25T18:26:46.870 INFO:teuthology.orchestra.run.smithi063.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40753.log.gz 2024-03-25T18:26:46.870 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.42006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40088.log 2024-03-25T18:26:46.870 INFO:teuthology.orchestra.run.smithi063.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41889.log.gz 2024-03-25T18:26:46.870 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39542.log: gzip -5 --verbose -- 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42006.log.gz 2024-03-25T18:26:46.870 INFO:teuthology.orchestra.run.smithi063.stderr: /var/log/ceph/ceph-client.admin.39581.log 2024-03-25T18:26:46.870 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40088.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39542.log.gz 2024-03-25T18:26:46.870 INFO:teuthology.orchestra.run.smithi063.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40088.log.gz 2024-03-25T18:26:46.870 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39581.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39581.log.gz 2024-03-25T18:26:46.870 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40323.log 2024-03-25T18:26:46.871 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40284.log 2024-03-25T18:26:46.871 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40206.log 2024-03-25T18:26:46.871 INFO:teuthology.orchestra.run.smithi063.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40323.log.gz 2024-03-25T18:26:46.871 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40792.log 2024-03-25T18:26:46.871 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39229.log 2024-03-25T18:26:46.871 INFO:teuthology.orchestra.run.smithi063.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40284.log.gz 2024-03-25T18:26:46.871 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40792.log: gzip -5 --verbose 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40206.log.gz 2024-03-25T18:26:46.871 INFO:teuthology.orchestra.run.smithi063.stderr: -- /var/log/ceph/ceph-client.admin.39150.log 2024-03-25T18:26:46.871 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39229.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40792.log.gz 2024-03-25T18:26:46.871 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39111.log 2024-03-25T18:26:46.871 INFO:teuthology.orchestra.run.smithi063.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39229.log.gz 2024-03-25T18:26:46.871 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41302.log 2024-03-25T18:26:46.871 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39111.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 86.5% 2024-03-25T18:26:46.871 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39150.log.gz 2024-03-25T18:26:46.872 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41302.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39111.log.gz 2024-03-25T18:26:46.872 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42279.log 2024-03-25T18:26:46.872 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-mgr.y.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41302.log.gz 2024-03-25T18:26:46.872 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40597.log 2024-03-25T18:26:46.872 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.42279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39346.log 2024-03-25T18:26:46.872 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40597.log: gzip -5 --verbose 86.1% -- /var/log/ceph/ceph-client.admin.41026.log 2024-03-25T18:26:46.872 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42279.log.gz 2024-03-25T18:26:46.872 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39346.log: gzip -5 --verbose -- 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40597.log.gz 2024-03-25T18:26:46.872 INFO:teuthology.orchestra.run.smithi063.stderr: /var/log/ceph/ceph-osd.0.log 2024-03-25T18:26:46.872 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41026.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39346.log.gz 2024-03-25T18:26:46.872 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41224.log 2024-03-25T18:26:46.872 INFO:teuthology.orchestra.run.smithi063.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41026.log.gz 2024-03-25T18:26:46.872 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40948.log 2024-03-25T18:26:46.872 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42240.log 2024-03-25T18:26:46.872 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40948.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41224.log.gz 2024-03-25T18:26:46.873 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41733.log 2024-03-25T18:26:46.873 INFO:teuthology.orchestra.run.smithi063.stderr:gzip 86.2% -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:26:46.873 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40948.log.gz 2024-03-25T18:26:46.873 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.42240.log: /var/log/ceph/ceph-client.admin.41733.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42240.log.gz 2024-03-25T18:26:46.873 INFO:teuthology.orchestra.run.smithi063.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41733.log.gz 2024-03-25T18:26:46.873 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:26:46.873 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39503.log 2024-03-25T18:26:46.873 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41772.log 2024-03-25T18:26:46.873 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39503.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39503.log.gz 2024-03-25T18:26:46.873 INFO:teuthology.orchestra.run.smithi063.stderr: 92.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:26:46.873 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39268.log 2024-03-25T18:26:46.873 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40519.log 2024-03-25T18:26:46.873 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41772.log: /var/log/ceph/ceph-client.admin.39268.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41772.log.gz 2024-03-25T18:26:46.873 INFO:teuthology.orchestra.run.smithi063.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39268.log.gz 2024-03-25T18:26:46.873 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39031.log 2024-03-25T18:26:46.874 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40519.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40519.log.gz 2024-03-25T18:26:46.874 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39737.log 2024-03-25T18:26:46.874 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39620.log 2024-03-25T18:26:46.874 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39737.log: /var/log/ceph/ceph-client.admin.39031.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39031.log.gz 2024-03-25T18:26:46.874 INFO:teuthology.orchestra.run.smithi063.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39737.log.gz 2024-03-25T18:26:46.874 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41655.log 2024-03-25T18:26:46.874 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39307.log 2024-03-25T18:26:46.874 INFO:teuthology.orchestra.run.smithi063.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:26:46.874 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39620.log: /var/log/ceph/ceph-client.admin.41655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39385.log 2024-03-25T18:26:46.874 INFO:teuthology.orchestra.run.smithi063.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39620.log.gz 2024-03-25T18:26:46.874 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39307.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41655.log.gz 2024-03-25T18:26:46.874 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42318.log 2024-03-25T18:26:46.874 INFO:teuthology.orchestra.run.smithi063.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39307.log.gz 2024-03-25T18:26:46.874 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41421.log 2024-03-25T18:26:46.875 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.42318.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39385.log.gz 2024-03-25T18:26:46.875 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39971.log 2024-03-25T18:26:46.875 INFO:teuthology.orchestra.run.smithi063.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42318.log.gz 2024-03-25T18:26:46.875 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5/var/log/ceph/ceph-client.admin.41421.log: --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:26:46.875 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39971.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41421.log.gz/var/log/ceph/ceph-mon.c.log: 2024-03-25T18:26:46.875 INFO:teuthology.orchestra.run.smithi063.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39971.log.gz 2024-03-25T18:26:46.919 INFO:teuthology.orchestra.run.smithi081.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:26:47.680 INFO:teuthology.orchestra.run.smithi063.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:26:47.697 INFO:teuthology.orchestra.run.smithi081.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:26:47.772 INFO:teuthology.orchestra.run.smithi081.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:26:47.962 INFO:teuthology.orchestra.run.smithi063.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:26:48.022 INFO:teuthology.orchestra.run.smithi081.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:26:48.034 INFO:teuthology.orchestra.run.smithi063.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:26:48.047 INFO:teuthology.orchestra.run.smithi081.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:26:48.065 INFO:teuthology.orchestra.run.smithi063.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:26:51.653 INFO:teuthology.orchestra.run.smithi063.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:26:51.750 INFO:teuthology.orchestra.run.smithi081.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:26:51.752 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T18:26:51.752 INFO:teuthology.orchestra.run.smithi081.stderr:real 0m4.929s 2024-03-25T18:26:51.752 INFO:teuthology.orchestra.run.smithi081.stderr:user 0m8.881s 2024-03-25T18:26:51.752 INFO:teuthology.orchestra.run.smithi081.stderr:sys 0m0.318s 2024-03-25T18:26:52.175 INFO:teuthology.orchestra.run.smithi063.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:26:52.178 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T18:26:52.178 INFO:teuthology.orchestra.run.smithi063.stderr:real 0m5.385s 2024-03-25T18:26:52.178 INFO:teuthology.orchestra.run.smithi063.stderr:user 0m14.220s 2024-03-25T18:26:52.178 INFO:teuthology.orchestra.run.smithi063.stderr:sys 0m0.512s 2024-03-25T18:26:52.179 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:26:52.179 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256/remote/smithi063/log 2024-03-25T18:26:52.180 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:26:54.597 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256/remote/smithi081/log 2024-03-25T18:26:54.598 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:26:56.159 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:26:56.400 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f1ace21d6739432fb6394ab83bf14564 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:26:56.404 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:26:56.414 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:26:56.422 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:26:56.423 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:26:56.437 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:26:56.518 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:26:56.527 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:26:56.527 DEBUG:teuthology.orchestra.run.smithi063:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:26:56.530 DEBUG:teuthology.orchestra.run.smithi081:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:26:56.544 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:26:56.547 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:26:56.549 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:26:56.549 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-03-25T18:26:56.549 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 63 +788us[ +786us] +/- 100ms 2024-03-25T18:26:56.550 INFO:teuthology.orchestra.run.smithi063.stdout:^* hv02.front.sepia.ceph.com 3 6 377 2 -1095us[-1097us] +/- 51ms 2024-03-25T18:26:56.550 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +933us[ +931us] +/- 72ms 2024-03-25T18:26:56.550 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:26:56.552 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:26:56.552 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-03-25T18:26:56.552 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 +815us[ +824us] +/- 101ms 2024-03-25T18:26:56.552 INFO:teuthology.orchestra.run.smithi081.stdout:^* hv02.front.sepia.ceph.com 3 6 377 2 -1088us[-1079us] +/- 51ms 2024-03-25T18:26:56.552 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +908us[ +917us] +/- 72ms 2024-03-25T18:26:56.552 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:26:56.553 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:26:56.562 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:26:56.563 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:26:56.572 DEBUG:teuthology.orchestra.run.smithi063:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:26:56.664 DEBUG:teuthology.orchestra.run.smithi081:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:26:56.735 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:26:56.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:26:56.767 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:26:56.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:26:56.798 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:26:56.808 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:26:56.817 INFO:teuthology.task.internal:Duration was 7965.473587 seconds 2024-03-25T18:26:56.817 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:26:56.826 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:26:56.826 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:26:56.828 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:26:56.870 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:26:56.882 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:26:57.235 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:26:57.235 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T18:26:57.235 DEBUG:teuthology.orchestra.run.smithi063:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:26:57.255 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T18:26:57.255 DEBUG:teuthology.orchestra.run.smithi081:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:26:57.283 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:26:57.283 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:26:57.297 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:26:57.351 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:26:57.352 DEBUG:teuthology.orchestra.run.smithi063:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:26:57.366 DEBUG:teuthology.orchestra.run.smithi081:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:26:57.494 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:26:57.504 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:26:57.504 DEBUG:teuthology.orchestra.run.smithi063:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:26:57.533 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:26:57.560 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:26:57.570 DEBUG:teuthology.orchestra.run.smithi063:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:26:57.573 DEBUG:teuthology.orchestra.run.smithi081:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:26:57.596 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = core 2024-03-25T18:26:57.626 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = core 2024-03-25T18:26:57.651 DEBUG:teuthology.orchestra.run.smithi063:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:26:57.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:26:57.676 DEBUG:teuthology.orchestra.run.smithi081:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:26:57.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:26:57.705 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:26:57.715 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:26:57.716 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256/remote/smithi063 2024-03-25T18:26:57.716 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:26:57.752 DEBUG:teuthology.misc:Transferring archived files from smithi081:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256/remote/smithi081 2024-03-25T18:26:57.753 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:26:57.788 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:26:57.789 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:26:57.791 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:26:57.837 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:26:57.848 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:26:57.849 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:26:57.857 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:26:57.858 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:26:57.860 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:26:57.872 INFO:teuthology.orchestra.run.smithi063.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:26 /home/ubuntu/cephtest 2024-03-25T18:26:57.892 INFO:teuthology.orchestra.run.smithi081.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:26 /home/ubuntu/cephtest 2024-03-25T18:26:57.893 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:26:57.903 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:26:57.922 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:26:57.946 DEBUG:teuthology.nuke:shortname: smithi063 2024-03-25T18:26:57.947 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:26:57.966 DEBUG:teuthology.nuke:shortname: smithi081 2024-03-25T18:26:57.966 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:26:57.969 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:05:36.330033', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCuqPn9P/7bTYjv4OwFl5It97HHvWJqNzhrbIGO9jXyoSczazWUGovbwr+XcXWyTjBXQq5Rnxziuri9JqQFfy2c='} 2024-03-25T18:26:57.985 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621256', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:05:36.328710', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBfBt3+3t0WojqxorpmvGW+mvVvJwbVcKMoUxRrsp1ZH3dMxNuK0Q3eqzhXGLQyLCJEcwaE+pFQXqyAv0TkZGFw='} 2024-03-25T18:26:58.023 INFO:teuthology.orchestra.console.smithi063:Power off 2024-03-25T18:26:58.024 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:26:58.047 DEBUG:teuthology.orchestra.console.smithi063:power off output: Chassis Power Control: Down/Off 2024-03-25T18:26:58.048 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:26:58.064 INFO:teuthology.orchestra.console.smithi081:Power off 2024-03-25T18:26:58.065 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:26:58.075 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-03-25T18:26:58.092 DEBUG:teuthology.orchestra.console.smithi081:power off output: Chassis Power Control: Down/Off 2024-03-25T18:26:58.092 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:26:58.114 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2024-03-25T18:27:02.077 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:27:02.118 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:27:02.188 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-03-25T18:27:02.231 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2024-03-25T18:27:06.192 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:27:06.232 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:27:06.303 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-03-25T18:27:06.343 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off 2024-03-25T18:27:06.444 INFO:teuthology.orchestra.console.smithi081:Power off completed 2024-03-25T18:27:10.306 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:27:10.416 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is off 2024-03-25T18:27:10.517 INFO:teuthology.orchestra.console.smithi063:Power off completed 2024-03-25T18:27:10.621 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7965.473587036133 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f1ace21d6739432fb6394ab83bf14564 status: fail success: false 2024-03-25T18:27:10.622 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:27:10.706 INFO:teuthology.run:FAIL